LLM Core API

API for calling LLM Core(s)

Setting Up the LLM Layer

The SDK provides a streamlined way to configure and initialize LLM components through the add_llm_layer method:

def add_llm_layer(self, config: LLMLayer) -> 'Cerebrum':
    """Set up the LLM core component."""
    result = self._post("/core/llm/setup", asdict(config))
    self._components_initialized.add("llm")
    self.results['llm'] = result
    return self

This method handles the LLM setup by sending the configuration to the /core/llm/setup endpoint, which is processed by the kernel.

LLM Configuration

The LLMLayer class defines the configuration parameters for the LLM:

@dataclass
class LLMLayer:
    llm_name: str                           # Name of the LLM model
    max_gpu_memory: dict | None = None      # GPU memory constraints
    eval_device: str = "cuda:0"             # Device for evaluation
    max_new_tokens: int = 2048              # Maximum token generation limit
    log_mode: str = "console"               # Logging configuration
    llm_backend: str = "default"            # Backend system selection

Note: Both llm_name and llm_backend are required parameters for all model types (open-source and closed-source). And close-sourced models can set up the GPU configurations by passing the max_gpu_memory to load the LLM and the eval_device to do specific inference.

Interacting with the LLM

Query Format

Interactions with the LLM require constructing a LLMQuery object:

class LLMQuery(Request):
    messages: List[Dict[str, Union[str, Any]]]    # Message list with role and content
    tools: Optional[List[Dict[str, Any]]] = Field(default_factory=list)  # Available tools
    action_type: Literal["chat", "tool_use", "operate_file"] = Field(default="chat")  # Action specification
    message_return_type: str = Field(default="text")  # Response format

    class Config:
        arbitrary_types_allowed = True

Response Format

The LLM's response follows the Response class structure:

class Response(BaseModel):
    response_message: Optional[str] = None  # Generated response text
    tool_calls: Optional[List[Dict[str, Any]]] = None  # Tool usage results
    finished: bool  # Completion status

    class Config:
        arbitrary_types_allowed = True