Quickstart

Be sure to complete the installation instructions before continuing with this guide.

Before starting AIOS, you need to make sure you have installed the LLM backends that you would like to run. Here are the LLM providers for supported backends for AIOS.

Provider 🏢	Model Name 🤖	Open Source 🔓	Model String ⌨️	Backend ⚙️	Required API Key
Anthropic	Claude 3.5 Sonnet	❌	claude-3-5-sonnet-20241022	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3.5 Haiku	❌	claude-3-5-haiku-20241022	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3 Opus	❌	claude-3-opus-20240229	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3 Sonnet	❌	claude-3-sonnet-20240229	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3 Haiku	❌	claude-3-haiku-20240307	anthropic	ANTHROPIC_API_KEY
Deepseek	Deepseek-reasoner (R1)	❌	deepseek-reasoner	deepseek	DEEPSEEK_API_KEY
Deepseek	Deepseek-chat (V3)	❌	deepseek-chat	deepseek	DEEPSEEK_API_KEY
OpenAI	GPT-4	❌	gpt-4	openai	OPENAI_API_KEY
OpenAI	GPT-4 Turbo	❌	gpt-4-turbo	openai	OPENAI_API_KEY
OpenAI	GPT-4o	❌	gpt-4o	openai	OPENAI_API_KEY
OpenAI	GPT-4o mini	❌	gpt-4o-mini	openai	OPENAI_API_KEY
OpenAI	GPT-3.5 Turbo	❌	gpt-3.5-turbo	openai	OPENAI_API_KEY
Google	Gemini 1.5 Flash	❌	gemini-1.5-flash	google	GEMINI_API_KEY
Google	Gemini 1.5 Flash-8B	❌	gemini-1.5-flash-8b	google	GEMINI_API_KEY
Google	Gemini 1.5 Pro	❌	gemini-1.5-pro	google	GEMINI_API_KEY
Google	Gemini 1.0 Pro	❌	gemini-1.0-pro	google	GEMINI_API_KEY
Groq	Llama 3.2 90B Vision	✅	llama-3.2-90b-vision-preview	groq	GROQ_API_KEY
Groq	Llama 3.2 11B Vision	✅	llama-3.2-11b-vision-preview	groq	GROQ_API_KEY
Groq	Llama 3.1 70B	✅	llama-3.1-70b-versatile	groq	GROQ_API_KEY
Groq	Llama Guard 3 8B	✅	llama-guard-3-8b	groq	GROQ_API_KEY
Groq	Llama 3 70B	✅	llama3-70b-8192	groq	GROQ_API_KEY
Groq	Llama 3 8B	✅	llama3-8b-8192	groq	GROQ_API_KEY
Groq	Mixtral 8x7B	✅	mixtral-8x7b-32768	groq	GROQ_API_KEY
Groq	Gemma 7B	✅	gemma-7b-it	groq	GROQ_API_KEY
Groq	Gemma 2B	✅	gemma2-9b-it	groq	GROQ_API_KEY
Groq	Llama3 Groq 70B	✅	llama3-groq-70b-8192-tool-use-preview	groq	GROQ_API_KEY
Groq	Llama3 Groq 8B	✅	llama3-groq-8b-8192-tool-use-preview	groq	GROQ_API_KEY
ollama	All Models	✅	model-name	ollama	-
vLLM	All Models	✅	model-name	vllm	-
HuggingFace	All Models	✅	model-name	huggingface	HF_HOME

Configuration

After installling LLM backends, multiple API keys may be required to set up. Here we provide the easier way to set up API keys, you can set up the following keys as below in your environment variables.

ANTHROPIC_API_KEY=''
DEEPSEEK_API_KEY=''
GEMINI_API_KEY=''
GROQ_API_KEY=''
HF_HOME=''
HF_AUTH_TOKENS=''
OPENAI_API_KEY=''

Recommended Configuration Method For simplicity and ease of use, we recommend using the config.yaml file to set up your API keys. This method is straightforward and helps avoid potential issues with environment variables. Simply add your API keys to the config.yaml file as shown below:

1. Set Up API Keys You need API keys for services like OpenAI, Anthropic, Groq and HuggingFace. The simplest way to configure them is to edit the aios/config/config.yaml.

2. Basic Configuration Add your API keys to the config.yaml. For example:

   openai: "your-openai-key"
   gemini: "your-gemini-key"
   deepseek: "your-deepseek-key"
   groq: "your-groq-key"
   anthropic: "your-anthropic-key"
   huggingface:
     auth_token: "your-huggingface-token"
     home: "optional-path"

3. Detailed Setup Instructions For detailed instructions on setting up API keys and configuration files, see Environment Variables Configuration.

How to Obtain API Keys

To obtain these API keys:

1. OpenAI API: Visit https://platform.openai.com/api-keys

2. Deepseek API: Visit https://api-docs.deepseek.com/

3. Google Gemini API: Visit https://aistudio.google.com/app/apikey

4. Groq API: Visit https://console.groq.com/keys

5. HuggingFace Token: Visit https://huggingface.co/settings/tokens

6. Anthropic API: Visit https://console.anthropic.com/keys

Use with ollama

You need to download ollama from from https://ollama.com/.

Then you need to start the ollama server either from ollama app or using the following command in the terminal.

$ ollama serve

Ollama can support both CPU-only and GPU environments. Details of how to use Ollama can be found at GitHub: Ollama.

Use with Huggingface Models

Some of the huggingface models require authentification, if you want to use all of the models you need to set up your authentification token in https://huggingface.co/settings/tokens and set up it as an environment variable using the following command

$ export HF_AUTH_TOKENS=<YOUR_TOKEN_ID>

By default, huggingface will download the models in the ~/.cache directory. If you want to designate the download directory, you can set up it using the following command

$ export HF_HOME=<YOUR_HF_HOME>

If you want to speed up the inference of huggingface models, you can use vLLM as the backend. The vLLM uses the exactly the same name huggingface native models.

It is important to note that vLLM currently only supports linux and GPU-enabled environment. So if you do not have the environment, you need to choose other model providers.

Considering that vLLM itself does not support passing designated GPU ids, you need to either setup the environment variable.

$ export CUDA_VISIBLE_DEVICES="0" # replace with your designated gpu ids

Launch AIOS

After you setup the required keys, you can run the following command to launch the AIOS kernel.

bash runtime/launch_kernel.sh

And then you can start a client to interact with the AIOS kernel using Terminal or WebUI.