Quickstart

Be sure to complete the installation instructions before continuing with this guide.

Before starting AIOS, you need to make sure you have installed the LLM backends that you would like to run. Here are the LLM providers for supported backends for AIOS.

Provider 🏢	Model Name 🤖	Open Source 🔓	Model String ⌨️	Backend ⚙️	Required API Key
Anthropic	Claude 3.5 Sonnet	❌	claude-3-5-sonnet-20241022	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3.5 Haiku	❌	claude-3-5-haiku-20241022	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3 Opus	❌	claude-3-opus-20240229	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3 Sonnet	❌	claude-3-sonnet-20240229	anthropic	ANTHROPIC_API_KEY
Anthropic	Claude 3 Haiku	❌	claude-3-haiku-20240307	anthropic	ANTHROPIC_API_KEY
OpenAI	GPT-4	❌	gpt-4	openai	OPENAI_API_KEY
OpenAI	GPT-4 Turbo	❌	gpt-4-turbo	openai	OPENAI_API_KEY
OpenAI	GPT-4o	❌	gpt-4o	openai	OPENAI_API_KEY
OpenAI	GPT-4o mini	❌	gpt-4o-mini	openai	OPENAI_API_KEY
OpenAI	GPT-3.5 Turbo	❌	gpt-3.5-turbo	openai	OPENAI_API_KEY
Google	Gemini 1.5 Flash	❌	gemini-1.5-flash	google	GEMINI_API_KEY
Google	Gemini 1.5 Flash-8B	❌	gemini-1.5-flash-8b	google	GEMINI_API_KEY
Google	Gemini 1.5 Pro	❌	gemini-1.5-pro	google	GEMINI_API_KEY
Google	Gemini 1.0 Pro	❌	gemini-1.0-pro	google	GEMINI_API_KEY
Groq	Llama 3.2 90B Vision	✅	llama-3.2-90b-vision-preview	groq	GROQ_API_KEY
Groq	Llama 3.2 11B Vision	✅	llama-3.2-11b-vision-preview	groq	GROQ_API_KEY
Groq	Llama 3.1 70B	✅	llama-3.1-70b-versatile	groq	GROQ_API_KEY
Groq	Llama Guard 3 8B	✅	llama-guard-3-8b	groq	GROQ_API_KEY
Groq	Llama 3 70B	✅	llama3-70b-8192	groq	GROQ_API_KEY
Groq	Llama 3 8B	✅	llama3-8b-8192	groq	GROQ_API_KEY
Groq	Mixtral 8x7B	✅	mixtral-8x7b-32768	groq	GROQ_API_KEY
Groq	Gemma 7B	✅	gemma-7b-it	groq	GROQ_API_KEY
Groq	Gemma 2B	✅	gemma2-9b-it	groq	GROQ_API_KEY
Groq	Llama3 Groq 70B	✅	llama3-groq-70b-8192-tool-use-preview	groq	GROQ_API_KEY
Groq	Llama3 Groq 8B	✅	llama3-groq-8b-8192-tool-use-preview	groq	GROQ_API_KEY
ollama	All Models	✅	model-name	ollama	-
vLLM	All Models	✅	model-name	vllm	-
HuggingFace	All Models	✅	model-name	huggingface	HF_HOME

Configuration

After installling LLM backends, multiple API keys may be required to set up. Here we provide the easier way to set up API API keys, you can create the .env file and set up the required keys and add new keys based on your needs.

OPENAI_API_KEY=''
GEMINI_API_KEY=''
HF_HOME=''
HF_AUTH_TOKENS=''
GROQ_API_KEY=''
ANTHROPIC_API_KEY=''

Use with OpenAI API

You need to get your OpenAI API key from https://platform.openai.com/api-keys. Then set up your OpenAI API key as an environment variable

$ export OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

Use with Google API

You need to get your Gemini API key from https://ai.google.dev/gemini-api. Then set up your Gemini API key as an environment variable.

$ export GEMINI_API_KEY=<YOUR_GEMINI_API_KEY>

Use with Anthropic API

You need to get your Anthropic API key from https://console.anthropic.com/settings/keys.

$ export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>

Use with ollama

You need to download ollama from from https://ollama.com/.

Then you need to start the ollama server either from ollama app or using the following command in the terminal.

$ ollama serve

To use models provided by ollama, you need to pull the available models from https://ollama.com/library

$ ollama pull llama3:8b # use llama3:8b for example

ollama can support CPU-only environment, in case you do not have CUDA environment.

Use with native huggingface models

Some of the huggingface models require authentification, if you want to use all of the models you need to set up your authentification token in https://huggingface.co/settings/tokens and set up it as an environment variable using the following command

$ export HF_AUTH_TOKENS=<YOUR_TOKEN_ID>

By default, huggingface will download the models in the ~/.cache directory. If you want to designate the download directory, you can set up it using the following command

$ export HF_HOME=<YOUR_HF_HOME>

Use with vLLM

If you want to speed up the inference of huggingface models, you can use vLLM as the backend. The vLLM uses the exactly the same name huggingface native models.

It is important to note that vLLM currently only supports linux and GPU-enabled environment. So if you do not have the environment, you need to choose other model providers.

Considering that vLLM itself does not support passing designated GPU ids, you need to either setup the environment variable.

$ export CUDA_VISIBLE_DEVICES="0" # replace with your designated gpu ids

Launch AIOS

After you setup the required keys, you can run the following command to launch the AIOS kernel.

bash runtime/launch_kernel.sh

And then you can start a client to interact with the AIOS kernel using Terminal or WebUI.