Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Microsoft's three-layer LLM routing architecture on AKS is directly relevant to AI agent orchestration and cloud infrastructure.
Microsoft's reference architecture for routing agent traffic on AKS uses three layers: RouteLLM for semantic routing (predicting when a cheaper model suffices), agentgateway as an OpenAI-compatible proxy for auth, rate limits, and cost tracking, and the Kubernetes Gateway API Inference Extension's Endpoint Picker for GPU-aware load balancing via vLLM metrics like KV-cache occupancy. RouteLLM's matrix-factorization router achieved ~95% of GPT-4 quality while sending only 26% of calls to the strong model, cutting costs by up to 85%, though Microsoft warns thresholds must be recalibrated per model pair and actual traffic. The design also integrates KAITO for GPU node pools and Azure Managed Prometheus/Grafana for unified observability of routing, cost, and GPU metrics.