LLM Core(s)

AIOS wraps different LLMs as LLM Cores to support the unified interface for addressing requests to both cloud LLM apis and locally-hosted LLMs.

AIOS supports three main categories of backends:

1. LiteLLM compatible backends (cloud and local)

2. vLLM backends (local)

3. Huggingface backends (local)

Each backend type handles different input scenarios, including standard text generation, tool calling, and JSON-formatted responses. This document explains how each backend processes these inputs and the implementation details.

Feature	LiteLLM Compatible	vLLM	Huggingface
Standard Input	Uses completion function	Uses OpenAI client	Uses generate method
Tool Calls	Native support via tools parameter	Native support via tools parameter	Uses message merging and custom decoding
JSON Responses	Uses format="json"	Uses format="json"	Uses message merging

Different backend details are as below

• Litellm compatible backends

• vLLM backends

• Huggingface local backends