Scheduler

Instead of placing separate queues within each processing module (e.g., LLM core(s) or memory manager), we centralize all the queues within the scheduler module. This aims to isolate the responsibility for request management from the individual modules, allowing each processing module to focus on its execution and simplifies the coordination of tasks across modules.

AIOS Scheduler manages the Request queue for each module, the Request queue includes both Queries (agent-to-kernel request) and Messages (module-to-module request)

The main purpose of the scheduler is to serve different agents queries in a fine-grained granularity (i.e., syscall level granuity) as below.

Scheduling example for syscalls for different agents (use LLM Syscall for illustration simplicity)

AIOS provides two main scheduler implementations to manage system calls from agents:

Feature	FIFOScheduler	RRScheduler
Task Selection	First-come, first-served	Round-robin rotation
Time Management	No time limit enforcement	Fixed time slices per task
Task Preemption	None (tasks run to completion)	Tasks can be suspended and resumed
Status Handling	Only "executing"/"done"	"executing"/"done"/"suspending"
Context Management	None	Uses SimpleContextManager

BaseScheduler

The BaseScheduler is an abstract base class that defines the common interface and functionality for all scheduler implementations in AIOS.

Abstract Methods

Method	Description
start_processing_threads()	Start threads for different request processors
stop_processing_threads()	Stop all processing threads gracefully
process_llm_requests()	Abstract method to process LLM requests
process_memory_requests()	Abstract method to process Memory requests
process_storage_requests()	Abstract method to process Storage requests
process_tool_requests()	Abstract method to process Tool requests
start()	Abstract method to start the scheduler
stop()	Abstract method to stop the scheduler

Different scheduler implementations

• FIFO Scheduler
• RR Scheduler