Routing agent traffic is really three decisions
A chat turn is one LLM call. An agent is hundreds — a reasoning loop that plans, calls a tool, reads the result, re-plans, and only sometimes stops. The cost and latency you signed up for in a demo get multiplied by that loop, and most of those calls are easy — a tool-argument fill, a yes/no gate, a short summary — that never needed a frontier model. So the question isn't "which model is best." It's "which model should answer this specific call, and how do I govern a flood of them?" That's not one decision; it's three, on different signals.
This post wires up those three decisions on AKS as a single OpenAI-compatible endpoint: a trivial agent step lands on a cheap self-hosted model (KAITO + vLLM, placed by GPU state) while a hard step escalates to a frontier model on Azure OpenAI — every call authenticated, metered, and traced. We lean on managed Azure throughout (KAITO for serving, Azure OpenAI for the frontier path, Azure Managed Prometheus and Grafana for observability); the two layers without a drop-in managed service — the semantic router and the AI gateway — run as open source on the cluster.
