When agents send LLM requests to AIOS kernel, the agent can choose one LLM backend or choose multiple LLM backends. 1) If only one LLM backend is specified, then the request will be sent to the specified LLM; 2) If multiple LLM backends are specified, it means that the agent allows this request to be processed by any of the specified LLMs. In this case, AIOS provides two LLM routing strategies — Sequential Routing and Smart Routing — to decide which LLM among the specified LLMs will be chosen to process the request.
Overview of Routing Strategies
AIOS provides two routing strategies to distribute requests across multiple LLM backends:
Strategy Type
Description
Sequential Routing
Sequentially cycles through the available models and selects an available one that is within the specific models
Smart Routing
A cost-quality optimized strategy which smartly chooses the lowest cost LLM while maintaining the quality of request processing
Sequential Routing
The SequentialRouting implements a basic model selection approach for load-balancing LLM requests. It sequentially cycles through the available models and selects an available models one that is within the specified models.
Core Functions
def get_model_idxs(self, selected_llms: List[str], n_queries: int=1):
model_idxs = []
for _ in range(n_queries):
current = selected_llms[self.idx]
for i, llm_config in enumerate(self.llm_configs):
if llm_config["name"] == current["name"]:
model_idxs.append(i)
break
self.idx = (self.idx + 1) % len(selected_llms)
return model_idxs
Smart Routing
The SmartRouting implements a sophisticated cost-quality optimized selection strategy for LLM requests, using historical performance data to predict which models will perform best for a given query while minimizing cost. It leverages a two-stage constrained optimization method. The figure below shows the overall pipeline of this smart routing strategy.
Optimization Methods
Uses Lagrangian dual optimization to globally optimize model selection
Balances overall performance against total cost
Ensures each query is assigned to exactly one model
def get_model_idxs(self, selected_llms, queries, input_token_lengths):
model_idxs = []
for i in range(len(queries)):
query = queries[i]
input_token_length = input_token_lengths[i]
# Get performance and length predictions
perf_scores, length_scores = self.store.predict(
query, selected_llms, n_similar=self.n_similar
)
# Calculate cost scores
cost_scores = []
for j in range(len(selected_llms)):
pred_output_length = length_scores[0][j]
input_cost = input_token_length * selected_llms[j].get("cost_per_input_token", 0)
output_cost = pred_output_length * selected_llms[j].get("cost_per_output_token", 0)
weighted_score = input_cost + output_cost
cost_scores.append(weighted_score)
# Select optimal model
selected_idx = self.optimize_model_selection(
selected_llms,
perf_scores[0],
np.array(cost_scores)
)
# Find index in original llm_configs
for idx, config in enumerate(self.llm_configs):
if config["name"] == selected_llms[selected_idx]["name"]:
model_idxs.append(idx)
break
return model_idxs
@article{mei2025eccos,
title={ECCOS: Efficient Capability and Cost Coordinated Scheduling for Multi-LLM Serving},
author={Mei, Kai and Xu, Wujiang and Lin, Shuhang and Zhang, Yongfeng},
journal={arXiv:2502.20576},
year={2025}
}
