This post wires up those three decisions on AKS as a single OpenAI-compatible endpoint: a trivial agent step lands on a cheap self-hosted model (KAITO + vLLM, placed by GPU state) while a hard step escalates to a frontier model on Azure OpenAI — every call authenticated, metered, and traced. We lean on managed Azure throughout (KAITO for serving, Azure OpenAI for the frontier path, Azure Managed Prometheus and Grafana for observability); the two layers without a drop-in managed service — the semantic router and the AI gateway — run as open source on the cluster.

The three decisions​

"Route this request" hides three separate questions, each reading a different signal, each with a different owner. For an agent firing calls in a loop, the first two fire on every call and the third on every weak-path call:

1. Which model should answer this call? A multi-step reasoning hop deserves a frontier model; "extract the date from this string" does not. Answering means reading the request content and predicting answer quality. This is semantic routing, and we'll use RouteLLM.
2. How do I expose, secure, govern, and forward this traffic? Provider abstraction, auth, per-agent budgets, rate limits, guardrails, observability. None of that cares about prompt meaning — it's a policy job for a data plane. We'll use agentgateway, an OpenAI-compatible, agent-native proxy, in the cluster.
3. Which replica of the chosen model gets this call? Across a fleet of GPU pods the right target depends on KV-cache occupancy, queue depth, and which LoRA adapters are already resident — live GPU state. That's the Gateway API Inference Extension and its Endpoint Picker.

Content → policy → GPU state. That's the through-line, and the discipline is refusing to let any layer answer a question that belongs to another: the router never thinks about auth, the gateway never thinks about prompt difficulty, the Endpoint Picker never thinks about which model — only which replica.

The architecture on AKS​

Most slots are managed Azure; where an open-source piece has a managed alternative, it's called out in the layer that earns it:

With the map in hand, the rest of this post builds it — bottom-up, serving first and the semantic decision last, because each layer needs the address of the one beneath it. First, the cluster everything runs on.

Prerequisites​

• An Azure subscription with GPU quota for the VM family you'll serve on (this guide uses a single Standard_NC* GPU node pool, provisioned for you by KAITO). Quota isn't capacity — confirm the SKU is actually allocatable in your region before you rely on it.
• A recent Azure CLI and kubectl.
• An Azure OpenAI (or AI Foundry) resource with a current frontier deployment for the strong path. We use gpt-5.1; gpt-4o is deprecating in many regions and may not be newly deployable.

Create a cluster with the serving and monitoring add-ons enabled. KAITO provisions and manages GPU node pools on demand, so you don't pre-create them:

az aks create \
  --resource-group $RG \
  --name $CLUSTER \
  --node-count 2 \
  --enable-ai-toolchain-operator \
  --enable-oidc-issuer \
  --enable-azure-monitor-metrics \
  --generate-ssh-keys

az aks get-credentials --resource-group $RG --name $CLUSTER

That's the whole cluster — no gateway add-on, no extra CRDs, no Istio to bootstrap. agentgateway is the front door (more on that in Layer 2). With the cluster up, start at the bottom: serving the weak model.

Layer 3 — Serving, and the load-balancing trap​

Start with the failure mode, because it's the whole reason this layer is more than a Service — and agents make it worse.

Round-robin assumes requests are interchangeable. LLM requests are anything but: a 100k-token context-stuffed agent step and a 200-token tool-argument fill differ by three orders of magnitude in cost. An agent fleet mixes both constantly. Spray them uniformly across identical pods and round-robin will cheerfully queue a cheap completion behind a giant prefill on a saturated pod while an idle pod sits next to it. Your throughput looks fine and your p99 — the thing an agent loop feels on every iteration — falls off a cliff.

The fix isn't more pods — it's placing each request on the pod best able to take it right now, by reading live KV-cache occupancy and queue depth. That capability is what the Endpoint Picker is. Everything else in this layer exists to feed it.

3.1 The pods to place: KAITO​

The Endpoint Picker needs a homogeneous pool of model servers exporting the right signals, and KAITO produces exactly that: declare a Workspace, point it at a preset, and KAITO provisions the GPU SKU, runs vLLM, and exposes an OpenAI-compatible Service. Because the engine is vLLM, every pod emits the metrics the Endpoint Picker routes on (vllm:num_requests_waiting, vllm:kv_cache_usage_perc). Apply kaito-workspace.yaml — it pins instanceType: Standard_NC24ads_A100_v4, count: 2, and preset: phi-4-mini-instruct:

kubectl create namespace llm
kubectl apply -f kaito-workspace.yaml
kubectl get workspace workspace-phi-4-mini -n llm -w   # Ready can take ~10–20 min

KAITO provisions the GPU node, pulls the model, and starts vLLM; the workspace flips to Ready when inference is up:

NAME                   INSTANCE                   RESOURCEREADY   INFERENCEREADY   STATE
workspace-phi-4-mini   Standard_NC24ads_A100_v4   True            True             Ready

KAITO publishes a Service named workspace-phi-4-mini on port 80 and labels its pods kaito.sh/workspace: workspace-phi-4-mini — that label is how the next step selects the pool. One detail that will bite you later: the Service maps port 80 to the vLLM container's port 5000, and the Endpoint Picker routes to pod IPs directly, bypassing the Service. So the pool below targets 5000, not 80.

3.2 The placement logic: the Inference Extension​

Pool membership — which pods are the model server — is a different concern from how a workload is weighted against that pool, so the Inference Extension splits them across two objects:

• An InferencePool — the set of model-server pods plus a reference to the Endpoint Picker (EPP), the extension that an ext-proc-capable proxy calls to choose an endpoint per request. In this design that proxy is agentgateway (Layer 2); the InferencePool and EPP don't care who calls them.
• An InferenceObjective — the per-workload treatment: an integer priority against a pool. It replaces the older InferenceModel object: where InferenceModel carried a modelName and a criticality enum, InferenceObjective keeps only an integer priority and takes the served-model name from the request instead.

Install the CRDs and the Endpoint Picker (the project ships a Helm chart), pointing it at the KAITO pods. The EPP must serve plaintext gRPC to match agentgateway's ext-proc client — set --secure-serving=false, or the call to it resets before headers:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.0/manifests.yaml

helm install phi-epp \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool \
  --version v1.0.0 -n llm \
  --set inferencePool.modelServers.matchLabels."kaito\.sh/workspace"=workspace-phi-4-mini \
  --set provider.name=none \
  --set inferenceExtension.extraArgs="{--secure-serving=false}"

The Helm chart creates its own InferencePool named after the release, so you don't strictly need to apply one. If you do, inference-pool.yaml has the full InferencePool + InferenceObjective — with two fields that trip people up. The pool's selector needs a structured matchLabels, and its targetPorts must be the vLLM container port 5000 (the pool routes to pod IPs, bypassing the Service's port 80):

spec:
  selector:
    matchLabels: { kaito.sh/workspace: workspace-phi-4-mini }
  targetPorts:
    - number: 5000          # the vLLM container port, NOT the Service's 80

The other trap is the API group: InferencePool lives in inference.networking.k8s.io, but InferenceObjective lives in inference.networking.x-k8s.io — different groups, easy to miss.

The Endpoint Picker is now running and watching the pool, but nothing calls it yet. It needs an ext-proc-capable proxy out front to consult it per request — and that proxy is the AI gateway in Layer 2, so we wire the call there rather than standing up a separate Gateway API gateway just to reach it.

What this layer decided: only which replica, on GPU state. It has no idea what a "strong" model is, no budgets, no auth. Those are the next layer's job.

Layer 2 — The gateway, and why agents need it most​

The tempting shortcut is to let the router call backends directly. Resist it — and with agents the reason is sharper than with chat. An agent loop can spend an unbounded number of tokens before it decides to stop; without a budget and a rate limit somewhere on the path, a single misbehaving agent can run up a frontier bill or saturate your GPU fleet. A gateway is where that policy lives, written once, so nothing else has to reinvent it.

agentgateway is an open-source, OpenAI-compatible AI gateway built for exactly this traffic: provider abstraction, auth, token-based rate limiting, model-cost tracking, guardrails, and OpenTelemetry across backends — and, because it's agent-native, first-class MCP tool-serving and agent-to-agent routing when you grow into them. Here it does three jobs: hide the strong (Azure OpenAI) and weak (in-cluster) backends behind one endpoint, enforce policy on every call, and — because it speaks ext-proc itself — call the Endpoint Picker directly so the weak path gets GPU-state placement. That third job is what lets us drop the Gateway API gateway entirely: one data plane instead of two, and no dependency on the managed gateway's inference-extension support, which is still preview-gated on AKS today.

The outer shape is binds → listeners → routes → backends. The one thing to internalize: an AI backend is ai: { name, provider: { <provider>: {...} } } — the provider nests under provider:, name is required, and the Azure provider needs resourceName and resourceType (its model is your Azure deployment name). Auth attaches as a route policy whose key the Azure provider sends as the api-key header (a static key keeps the example self-contained; for production on AKS, prefer Azure workload identity, which agentgateway supports natively for the Azure provider):

# the strong route's backend (excerpt)
backends:
  - ai:
      name: aoai-strong
      provider:
        azure:
          resourceName: "<your-aoai-resource>"
          resourceType: openAI
          model: gpt-5.1
          apiVersion: "2024-10-21"

The strong route is a single ai backend pointed at Azure OpenAI. The weak route is where the front-door magic lives: a service backend pointed at the KAITO pods, with an inferenceRouting policy that hands each request to the Endpoint Picker over ext-proc and forwards to the pod it returns. Three details earned during validation, because the loader is strict about each:

# top-level: declare the model-server pool the picker selects from
services:
  - name: phi-weak
    namespace: llm
    hostname: workspace-phi-4-mini.llm.svc.cluster.local
    vips: []                       # required, even when empty
    ports: { 5000: 5000 }          # the vLLM container port

# ...the weak route's backend:
backends:
  - service:
      name: llm/workspace-phi-4-mini.llm.svc.cluster.local   # namespace/HOSTNAME, not the service name
      port: 5000
    policies:
      inferenceRouting:            # a BACKEND policy, not a route policy
        endpointPicker: { host: phi-epp-epp.llm.svc.cluster.local:9002 }
        destinationMode: passthrough   # trust the picker's choice

The full two-route config — including the strong ai backend and both routes' policies — is in agentgateway-config.yaml. The credential never leaves the gateway, and on the weak route agentgateway itself is the thing that consults the picker.

The governance that makes this safe for agents lives there too: the strong route carries a token rate limit (a localRateLimit route policy) — the backstop against an agent loop that won't stop spending. agentgateway can also price every call in real dollars through a top-level config.modelCatalog, but its schema is nested and provider-scoped — the obvious flat form didn't load for us — so wire it from the agentgateway docs for your release rather than copying a snippet.

Smoke-test both paths before the router exists:

# strong → Azure OpenAI
curl -s http://agentgateway.llm.svc.cluster.local:4000/strong/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.1","messages":[{"role":"user","content":"Say no in one word."}]}'

# weak → KAITO via the Endpoint Picker
# (same shape RouteLLM sends: it appends /v1/chat/completions to api_base)
curl -s http://agentgateway.llm.svc.cluster.local:4000/weak/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"phi-4-mini-instruct","messages":[{"role":"user","content":"Say hi in one word."}]}'

Both return an OpenAI-shaped completion — the strong call comes back stamped with the resolved Azure deployment, the weak call from the in-cluster model, and neither caller had to know where the model lived:

// strong → Azure OpenAI
{ "model": "gpt-5.1-2025-11-13",
  "choices": [{ "message": { "role": "assistant", "content": "No." } }],
  "usage": { "total_tokens": 29 } }

// weak → KAITO / vLLM
{ "model": "phi-4-mini-instruct",
  "choices": [{ "message": { "role": "assistant", "content": "Hello." } }] }

What this layer decided: how to authenticate, meter, guardrail, and forward — on policy, never on content. It dispatches whatever model name it's handed. Choosing that name is Layer 1's job.

Layer 1 — The semantic decision, and the threshold that is the whole ballgame​

Here's the only layer that reads the prompt — and the one decision that actually moves the bill. For an agent firing hundreds of calls per task, it's also the layer with the most leverage: every easy call it keeps off the frontier model is multiplied by the loop.

RouteLLM is a set of routers trained on human-preference data to predict, per prompt, whether a cheap model can match a strong model's answer. On the paper's own model pair, the featured mf (matrix-factorization) router reaches ~95% of GPT-4's quality on MT-Bench while making just ~26% of the GPT-4 calls — cutting cost by up to ~85% versus sending everything to GPT-4, by escalating only the genuinely hard prompts.

The composition is the elegant part: RouteLLM's strong and weak upstreams are just agentgateway. It makes the semantic call, picks a model, and forwards into the governed data plane — so the decision layer inherits all of Layer 2's auth, budgets, and tracing without re-implementing a line of it.

One wiring detail the reviewer in you should catch: agentgateway routes by path (/strong, /weak), but RouteLLM hands its choice to litellm as a model name. So point each model at the matching agentgateway path with a per-model api_base rather than a single base URL:

# inside the RouteLLM container — each model maps to its agentgateway route
python -m routellm.openai_server --routers mf \
  --strong-model openai/gpt-5.1 \
  --weak-model   openai/phi-4-mini-instruct \
  --config routellm-config.yaml   # sets api_base per model → /strong and /weak

One more gotcha from running this against current models: RouteLLM sends presence_penalty / frequency_penalty by default, and litellm (which RouteLLM uses to reach each provider) raises UnsupportedParamsError when the target model doesn't accept them — newer frontier models like the gpt-5.x family don't. Set litellm.drop_params=True so litellm silently drops the unsupported params instead of failing the call.

Clients call RouteLLM with a model string that encodes both the router and the cost threshold:

router-mf-0.11593
        │   └── threshold: the strong/weak cutoff — calibrate it
        └────── the matrix-factorization router

That threshold is the ballgame. Set it too low and you escalate everything and pay frontier prices for "hello"; set it too high and quality craters on the hard steps that needed the strong model. Don't guess it — calibrate against a representative sample so a target fraction of your traffic escalates:

python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5
# → 0.11593: the threshold that sends ~50% of calls to "strong"

Then close the loop with the real numbers. Run live agent traffic and read the actual strong/weak split off agentgateway's metrics — not RouteLLM's estimate — and, once you've wired the cost catalog, the dollars too; nudge the threshold until the cost/quality trade sits where you want it. The router proposes; your own traffic disposes.

One caveat the rate card can't see: prompt caching. A cached input token costs a fraction of an uncached one — Azure OpenAI discounts cache hits, and the weak path's vLLM prefix cache is the very thing the Endpoint Picker routes on. Both caches are per-model and per-prefix, so per-call routing works against them: bounce a session between models and each one's cache for that growing context goes cold, so the marginal cost of a strong call whose prefix is still warm is much lower than its sticker rate, and the savings from offloading it are smaller than the headline prices imply. This is another reason to measure billed cost rather than multiply tokens by a static rate — and, where caching dominates over difficulty, to route at session or task granularity instead of per call. Per-call routing on prompt difficulty is still the right default; just don't let a token-rate estimate convince you the split is cheaper or more expensive than your bill says.

Observability with managed Azure monitoring​

Layer 1 trades quality for cost, so this section closes one question: can you trust the routing numbers, and can you watch the same signals the Endpoint Picker routes on? The answer splits by source — GPU and latency come from vLLM; the strong/weak split (and, once you wire the cost catalog, dollars) comes from agentgateway — so a complete view scrapes both.

To get there, add two PodMonitors to the Managed Prometheus you enabled at cluster creation: one selecting the KAITO pods for vLLM's metrics on port 5000, one for agentgateway's on port 15020. Then link Azure Managed Grafana to the workspace. (One gotcha: the managed add-on uses Azure's CRDs under the azmonitoring.coreos.com API group, not upstream monitoring.coreos.com, so change a community manifest's apiVersion to azmonitoring.coreos.com/v1 or the add-on won't scrape it.)

The full path, one call​

# trivial step → router picks weak → agentgateway → Endpoint Picker → KAITO pod
curl -s http://routellm.llm.svc.cluster.local:6060/v1/chat/completions \
  -d '{"model":"router-mf-0.11593","messages":[{"role":"user","content":"What is 2+2?"}]}'

# hard step → router picks strong → agentgateway → Azure OpenAI
curl -s http://routellm.llm.svc.cluster.local:6060/v1/chat/completions \
  -d '{"model":"router-mf-0.11593",
       "messages":[{"role":"user","content":"Derive the EM algorithm and explain why each step cannot decrease the log-likelihood."}]}'

The model field in each response is the receipt — it tells you which way the router decided. Against the calibrated threshold 0.11593, "What is 2+2?" scores a strong-win-rate of 0.09 (below the line → weak) while the EM-algorithm prompt scores 0.29 (above → strong), so they come back from different models:

// "What is 2+2?"             → "model": "phi-4-mini-instruct"   (weak, a fraction of a cent)
// "Derive the EM algorithm…" → "model": "gpt-5.1-2025-11-13"    (strong, frontier pricing)

Both hit the same endpoint with the same model string. Three decisions — content, policy, GPU state — resolved in one round trip, each by the layer that owns it. Now point your agent framework's base URL at that endpoint and every call in the loop gets sorted the same way.

Scale it to your problem​

You rarely need all of it on day one. Because the layers don't bleed into each other, you can adopt exactly the slice your problem demands:

• One hosted model, a few agents, no self-hosting? Layer 2 only — agentgateway for auth, budgets, and guardrails over Azure OpenAI. There's nothing to route to and no GPUs to place on.
• Self-hosting at scale but one model class? KAITO + the Inference Extension give you managed serving and cache-aware placement. Skip semantic routing — there's only one model.
• A real cost problem from agents sending easy steps to expensive models? Add RouteLLM. It earns its keep the moment you have a meaningful price gap and a meaningful fraction of easy traffic — which describes essentially every agent in a loop.

Conclusion​

Routing agent traffic isn't one decision — it's three, each on its own signal: content picks the model (RouteLLM), policy governs the call (agentgateway), and GPU state picks the replica (the Inference Extension over KAITO-served vLLM). Keep those seams clean and you get a single OpenAI-compatible endpoint your agents point at, where every call is routed, metered, and traced by the layer that owns that concern — and where the easy calls an agent loop fires by the hundred quietly stay off the frontier bill.

From here, the obvious next moves all reinforce the same shape: autoscale the weak fleet on vllm:num_requests_waiting with the managed KEDA add-on so the GPU pool breathes with bursty agent traffic; lean into agentgateway's agent-native side by serving tools over MCP and routing agent-to-agent traffic through the same governed data plane; and add per-agent budgets at the gateway so spend stays attributable as more agents share the cluster.

The example manifests have everything here ready to adapt. For the components themselves, see KAITO on AKS, the Gateway API Inference Extension, agentgateway, RouteLLM (and its paper), and Managed Prometheus and Grafana for AKS.

The official quickstart walks you through deploying Anyscale on Azure using the Azure CLI and Azure Portal with Envoy Gateway for ingress. That's a solid starting point, but if you're like me, you want everything in Terraform so your infrastructure is repeatable, version-controlled, and easy to tear down.

In this post, we'll walk through a Terraform configuration that deploys the full Anyscale on Azure stack, swapping out Envoy Gateway for AKS managed Gateway API with Istio-based app routing. With this approach, AKS handles the gateway lifecycle and you skip the step of installing and configuring Envoy Gateway entirely.

What we're building​

Before we dive into the code, here's a quick overview of what the Terraform configuration creates:

• A resource group to hold everything
• An AKS cluster (Standard tier) with Gateway API and Istio-based app routing enabled
• The Anyscale operator installed as an AKS cluster extension
• A user-assigned managed identity with a federated credential for the Anyscale operator
• An Azure Container Registry (Standard SKU) for container images
• A storage account with hierarchical namespace (HNS) enabled and a private blob container
• The Anyscale Cloud and Cloud Resource registered via the AzAPI provider (Anyscale.Platform/clouds)
• Role assignments for storage, container registry, and AKS access

The full Terraform code is available as a GitHub Gist. In this post, I'll walk through each piece so you understand what's happening and why.

Prerequisites​

Before you get started, make sure you have the following:

• Azure CLI authenticated to a subscription
• Terraform
• Anyscale CLI (installed via pipx)
• jq for parsing Terraform outputs
• kubectl for interacting with the AKS cluster

Your subscription also needs the following resource providers registered:

• Anyscale.Platform
• Microsoft.ContainerService
• Microsoft.ContainerRegistry
• Microsoft.Storage
• Microsoft.ManagedIdentity

Step 1: Set up the Terraform providers​

Create a new directory for your Terraform configuration and create a file named main.tf. For simplicity, we'll put everything in a single file so it's easy to follow along. In a real project, you'd probably want to split things up into separate files. See the Terraform style guide on file names for best practices.

We'll need a few providers: azurerm for Azure resources, azapi for the Anyscale Platform resources, azuread for the service principal lookup, and a couple of utility providers. Add the following to your main.tf:

terraform {
  required_providers {
    azapi = {
      source  = "Azure/azapi"
      version = "~> 2.10.0"
    }

    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.8.0"
    }

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.76.0"
    }

    local = {
      source  = "hashicorp/local"
      version = "~> 2.9.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.9.0"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

We're using azapi here because the Anyscale.Platform resource type is brand new and not yet in the azurerm provider. The azapi provider lets us work with any ARM resource type directly, including preview ones. At the time of this writing, the Anyscale.Platform resource provider is not yet fully documented. I'll go into how I figured out the API schema for undocumented resources like this when we get to Step 5.

We also set prevent_deletion_if_contains_resources = false on the azurerm provider so that terraform destroy can cleanly remove the resource group without getting blocked by the resources inside it.

Step 2: Create the foundational resources​

Let's start with the basics: a resource group, some random naming helpers, and the storage and container registry resources that Anyscale needs. Add the following to your main.tf:

variable "location" {
  description = "Azure region. Must support Anyscale on Azure."
  type        = string
  default     = "westus3"
  validation {
    condition     = contains(["westcentralus", "eastus", "eastus2", "westus2", "westus3", "southcentralus"], var.location)
    error_message = "Must be a region that supports Anyscale on Azure."
  }
}

data "azurerm_client_config" "current" {}

resource "random_integer" "example" {
  min = 10
  max = 99
}

resource "random_string" "example" {
  length  = 4
  special = false
  upper   = false
  lower   = true
  numeric = false
}

locals {
  random_name = "anyscale${random_integer.example.result}"
}

resource "azurerm_resource_group" "example" {
  name     = "rg-${local.random_name}"
  location = var.location
}

The location variable has a validation block that restricts you to regions where Anyscale on Azure is currently supported. This saves you from potential deployment failures later.

Next, add the container registry and storage account:

resource "azurerm_container_registry" "example" {
  name                = "${local.random_name}${random_string.example.result}"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "Standard"
  admin_enabled       = false

  tags = {
    "anyscale-cloud" = local.random_name
  }
}

resource "azurerm_storage_account" "example" {
  name                            = "${local.random_name}${random_string.example.result}"
  resource_group_name             = azurerm_resource_group.example.name
  location                        = azurerm_resource_group.example.location
  account_kind                    = "StorageV2"
  account_tier                    = "Standard"
  account_replication_type        = "LRS"
  access_tier                     = "Hot"
  min_tls_version                 = "TLS1_2"
  https_traffic_only_enabled      = true
  is_hns_enabled                  = true
  default_to_oauth_authentication = true
  shared_access_key_enabled       = true

  blob_properties {
    cors_rule {
      allowed_headers    = ["*"]
      allowed_methods    = ["DELETE", "GET", "HEAD", "POST", "PUT"]
      allowed_origins    = ["https://*.anyscale.com"]
      exposed_headers    = ["*"]
      max_age_in_seconds = 3600
    }
  }

  tags = {
    "anyscale-cloud" = local.random_name
  }
}

resource "azurerm_storage_container" "example" {
  name                  = "anyscale-data"
  storage_account_id    = azurerm_storage_account.example.id
  container_access_type = "private"
}

A few things to note here. The storage account has is_hns_enabled = true because Anyscale uses Azure Data Lake Storage Gen2 (ADLS Gen2) for its cloud storage. HNS gives you hierarchical file system semantics on top of blob storage, which is what the abfss:// URI scheme expects. The CORS rule allows the Anyscale web console to interact with the storage account directly from the browser.

Step 3: Create the AKS cluster with Gateway API​

This is where things get interesting. Instead of using the azurerm_kubernetes_cluster resource, we're using azapi_resource to create the AKS cluster. The managed Gateway API and Istio-based app routing features also went GA recently, but the ability to configure these features hasn't landed in the azurerm provider yet. So we use azapi to hit the AKS API directly. Add the following to your main.tf:

resource "azapi_resource" "aks" {
  type      = "Microsoft.ContainerService/managedClusters@2026-03-02-preview"
  parent_id = azurerm_resource_group.example.id
  location  = azurerm_resource_group.example.location
  name      = "aks-${local.random_name}"

  schema_validation_enabled = false

  body = {
    identity = {
      type = "SystemAssigned"
    },
    properties = {
      dnsPrefix = "aks-${local.random_name}"
      agentPoolProfiles = [
        {
          name              = "systempool"
          mode              = "System"
          enableAutoScaling = true
          minCount          = 3
          maxCount          = 6
        }
      ]
      ingressProfile = {
        gatewayAPI = {
          installation = "Standard"
        }
        webAppRouting = {
          gatewayAPIImplementations = {
            appRoutingIstio = {
              mode = "Enabled"
            }
          }
        }
      }
      oidcIssuerProfile = {
        enabled = true
      }
      securityProfile = {
        workloadIdentity = {
          enabled = true
        }
      }
    }
    sku = {
      name = "Base"
      tier = "Standard"
    }
  }

  response_export_values = [
    "properties.oidcIssuerProfile.issuerURL",
    "properties.identityProfile.kubeletidentity.objectId"
  ]
}

Let's break down the key parts of the ingressProfile:

• gatewayAPI.installation = "Standard" installs the Gateway API CRDs and controller on the cluster
• webAppRouting.gatewayAPIImplementations.appRoutingIstio.mode = "Enabled" enables the Istio-based implementation, which gives you the approuting-istio gateway class

We also enable OIDC issuer and workload identity on the cluster. These are required for the Anyscale operator to authenticate with Azure services using federated credentials instead of storing secrets.

The response_export_values block tells azapi to capture the OIDC issuer URL and the kubelet identity object ID from the cluster response, which we'll need for the federated credential and role assignments later.

Step 4: Set up identity and role assignments​

The Anyscale operator needs a managed identity with specific permissions to access resources on your behalf. We'll create a user-assigned managed identity, set up a federated credential so it can authenticate from within the cluster, and assign the necessary roles. Add the following to your main.tf:

resource "azuread_service_principal" "example" {
  client_id    = "086bc555-6989-4362-ba30-fded273e432b" # Anyscale Kubernetes Operator Auth
  use_existing = true
}

resource "azurerm_user_assigned_identity" "example" {
  location            = azurerm_resource_group.example.location
  name                = "${local.random_name}-operator-identity"
  resource_group_name = azurerm_resource_group.example.name

  tags = {
    "anyscale-cloud" = local.random_name
  }
}

resource "azurerm_federated_identity_credential" "example" {
  name                      = azurerm_user_assigned_identity.example.name
  audience                  = ["api://AzureADTokenExchange"]
  issuer                    = azapi_resource.aks.output.properties.oidcIssuerProfile.issuerURL
  user_assigned_identity_id = azurerm_user_assigned_identity.example.id
  subject                   = "system:serviceaccount:anyscale-operator:anyscale-operator"
}

The azuread_service_principal resource references the well-known Anyscale Kubernetes Operator Auth application in your tenant. The use_existing = true flag tells Terraform to look up the existing service principal rather than trying to create one. This is needed so the Anyscale operator can authenticate against your cluster.

The federated identity credential is the glue between the Kubernetes service account (anyscale-operator in the anyscale-operator namespace) and the Azure managed identity. When the operator pod starts, it can exchange its Kubernetes service account token for an Azure access token without needing any stored secrets.

Now add the role assignments:

resource "azurerm_role_assignment" "acr_pull" {
  principal_id                     = azapi_resource.aks.output.properties.identityProfile.kubeletidentity.objectId
  role_definition_name             = "AcrPull"
  scope                            = azurerm_container_registry.example.id
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "blob_data_owner" {
  principal_id                     = azurerm_user_assigned_identity.example.principal_id
  role_definition_name             = "Storage Blob Data Owner"
  scope                            = azurerm_storage_account.example.id
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "acr_push" {
  principal_id                     = azurerm_user_assigned_identity.example.principal_id
  role_definition_name             = "AcrPush"
  scope                            = azurerm_container_registry.example.id
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "acr_task_contributor" {
  principal_id                     = azurerm_user_assigned_identity.example.principal_id
  role_definition_name             = "Container Registry Tasks Contributor"
  scope                            = azurerm_container_registry.example.id
  skip_service_principal_aad_check = true
}

Here's what each role does:

• AcrPull for the kubelet identity so the cluster can pull container images
• Storage Blob Data Owner for the operator identity to read/write data in the storage account
• AcrPush and Container Registry Tasks Contributor for the operator identity to push images and run ACR tasks

Step 5: Register the Anyscale Cloud resources​

Now we register the Anyscale Cloud and Cloud Resource using the Anyscale.Platform resource provider through azapi. These resources tell the Anyscale control plane about your Azure environment.

Add the following to your main.tf:

resource "azapi_resource" "anyscale_cloud" {
  type      = "Anyscale.Platform/clouds@2026-02-01-preview"
  parent_id = azurerm_resource_group.example.id
  location  = azurerm_resource_group.example.location
  name      = local.random_name

  schema_validation_enabled = false

  body = {
    properties = {
      acrResourceId = azurerm_container_registry.example.id
    }
  }

  tags = {
    "anyscale-cloud" = local.random_name
  }

  response_export_values = [
    "properties.ssoUrl",
    "properties.cloudResourceId"
  ]

  depends_on = [
    azapi_resource.aks
  ]
}

resource "azapi_resource" "anyscale_cloud_resource" {
  type      = "Anyscale.Platform/clouds/cloudResources@2026-02-01-preview"
  parent_id = azapi_resource.anyscale_cloud.id
  location  = azurerm_resource_group.example.location
  name      = "default"

  schema_validation_enabled = false

  body = {
    properties = {
      provider                    = "Azure"
      computeStack                = "K8S"
      cloudStorageBucketEndpoint  = azurerm_storage_account.example.primary_blob_endpoint
      cloudStorageBucketName      = "abfss://${azurerm_storage_container.example.name}@${azurerm_storage_account.example.primary_dfs_host}"
      anyscaleOperatorIamIdentity = azurerm_user_assigned_identity.example.principal_id
    }
  }

  tags = {
    "anyscale-cloud" = local.random_name
  }

  response_export_values = [
    "properties.cloudResourceId"
  ]

  depends_on = [
    azapi_resource.aks
  ]
}

The Anyscale.Platform/clouds resource registers a new Anyscale "cloud" (their term for a connected infrastructure environment) and associates it with your container registry. The child cloudResources resource provides the details about compute (Kubernetes), storage (the ADLS Gen2 bucket), and the operator identity.

Notice the cloudStorageBucketName uses the abfss:// URI format. This is the Azure Blob File System (ABFS) scheme that ADLS Gen2 uses, and it's how Anyscale accesses your storage.

Step 6: Install the Anyscale operator extension​

The Anyscale operator is deployed as an AKS cluster extension. This is where we wire up the gateway configuration. Add the following to your main.tf:

resource "azurerm_kubernetes_cluster_extension" "anyscale_operator" {
  name           = "anyscaleoperator"
  cluster_id     = azapi_resource.aks.id
  extension_type = "Anyscale.AKS.Operator"
  release_train  = "stable"

  plan {
    name      = "anyscale-operator"
    product   = "anyscale-operator-aks"
    publisher = "anyscale1750870039553"
  }

  configuration_settings = {
    "global.auth.audience"          = "api://086bc555-6989-4362-ba30-fded273e432b/.default"
    "global.auth.iamIdentity"       = azurerm_user_assigned_identity.example.client_id
    "global.cloudDeploymentId"      = azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId
    "global.controlPlaneURL"        = "https://console.azure.anyscale.com"
    "workloads.serviceAccount.name" = "anyscale-operator"
    "networking.gateway.enabled"    = "true"
    "networking.gateway.name"       = "gateway"
    "networking.gateway.className"  = "approuting-istio"
    "networking.gateway.namespace"  = "anyscale-operator"
    "networking.gateway.apiVersion" = "gateway.networking.k8s.io/v1"
    "networking.gateway.hostname"   = "${replace(azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId, "_", "-")}.${azapi_resource.aks.location}.cloudapp.azure.com"
  }

  depends_on = [
    azurerm_federated_identity_credential.example
  ]
}

The key configuration settings to pay attention to:

• networking.gateway.className = "approuting-istio" tells the operator to use the AKS managed Istio gateway class
• networking.gateway.apiVersion = "gateway.networking.k8s.io/v1" ensures we're using the GA Gateway API spec
• networking.gateway.hostname constructs a DNS label that Azure will use to assign a public FQDN to the gateway's load balancer

The plan block references the Azure Marketplace listing for the Anyscale operator. This is how Azure knows what extension to install and which publisher it comes from.

We also need a role assignment so you can manage the Anyscale platform resources. Add this to your main.tf:

resource "azurerm_role_assignment" "anyscale_platform_contributor" {
  principal_id         = data.azurerm_client_config.current.object_id
  role_definition_name = "Anyscale Platform Contributor Role"
  scope                = azapi_resource.anyscale_cloud.id
}

Step 7: Outputs and the Gateway template​

Before we deploy, we need two more things: Terraform output values so we can grab resource names after the deployment, and a template file for the Gateway manifest.

First, add the local_file resource to your main.tf. It takes a template (which we'll create next) and renders it into anyscale-gateway.yaml with the actual cloud resource ID:

resource "local_file" "anyscale_gateway" {
  filename = "anyscale-gateway.yaml"
  content = templatefile("anyscale-gateway.tmpl",
    {
      ANYSCALE_CLOUD_RESOURCE_ID = replace(azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId, "_", "-")
    }
  )
}

Next, add the following outputs to your main.tf:

output "rg_name" {
  value = azurerm_resource_group.example.name
}

output "aks_name" {
  value = azapi_resource.aks.name
}

output "anyscale_cloud_id" {
  value = split("/", azapi_resource.anyscale_cloud.output.properties.ssoUrl)[4]
}

output "anyscale_cloud_resource_id" {
  value = azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId
}

Finally, create a file named anyscale-gateway.tmpl in the same directory. Terraform uses this template to generate the Gateway manifest with your cloud resource ID injected:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: anyscale-operator
spec:
  gatewayClassName: approuting-istio
  infrastructure:
    annotations:
      service.beta.kubernetes.io/azure-dns-label-name: ${ANYSCALE_CLOUD_RESOURCE_ID}
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Same
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "*.i.azure.anyscaleuserdata.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: anyscale-${ANYSCALE_CLOUD_RESOURCE_ID}-certificate
      allowedRoutes:
        namespaces:
          from: Same
    - name: https-session
      port: 443
      protocol: HTTPS
      hostname: "*.s.azure.anyscaleuserdata.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: anyscale-${ANYSCALE_CLOUD_RESOURCE_ID}-certificate
      allowedRoutes:
        namespaces:
          from: Same

The Gateway defines three listeners: one for HTTP traffic and two for HTTPS. The HTTPS listeners handle Anyscale's interactive (*.i.) and session (*.s.) subdomains with TLS termination. The azure-dns-label-name annotation tells Azure to assign a public DNS label to the gateway's load balancer IP.

Deploy it​

With all the Terraform code in place, it's time to deploy. Sign in to Azure and set your subscription:

az login
az account set -s <subscription-id>
export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)

Initialize Terraform to download the providers:

terraform init

Then format and validate to make sure everything is clean:

terraform fmt
terraform validate

If validation passes, go ahead and apply:

terraform apply

Review the plan and type yes to proceed. The deployment takes about 7-10 minutes since it's creating an AKS cluster, installing the Anyscale operator extension, and registering the cloud resources.

Once it's done, grab the outputs and connect to the cluster:

read -r \
  RG_NAME \
  AKS_NAME \
  ANYSCALE_CLOUD_ID \
  ANYSCALE_CLOUD_RESOURCE_ID <<< "$(terraform output -json | jq -r \
    '[.rg_name.value,
      .aks_name.value,
      .anyscale_cloud_id.value,
      .anyscale_cloud_resource_id.value] | @tsv')"

az aks get-credentials -g $RG_NAME -n $AKS_NAME

Then deploy the gateway:

kubectl apply -f anyscale-gateway.yaml

Verify the deployment​

Check that the Anyscale operator and gateway-related pods are running:

kubectl get po -A

You should see pods in the anyscale-operator namespace for the operator itself, and Istio-related pods in the gateway infrastructure namespaces.

Now sign in to the Anyscale console. I installed the Anyscale CLI using pipx, which I like because it installs Python CLI tools in isolated environments, similar to how npx works in the Node.js world. You get the tool without polluting your global Python packages.

pipx install anyscale
export ANYSCALE_HOST=https://console.azure.anyscale.com
anyscale login

Verify the cloud is registered and healthy:

anyscale cloud list
anyscale cloud verify --id $ANYSCALE_CLOUD_ID

The cloud list command should show your cloud with the AZURE provider and your region. The cloud verify command runs a series of checks against your cluster, including operator health, identity, and networking. You want to see all checks come back as PASSED.

From here, you can follow the quickstart guide to run your first workload. You can also open the Anyscale console directly from the Azure portal by navigating to your Anyscale resource and selecting the console link. That's where you can explore workspaces, submit jobs, and manage your Ray clusters.

Cleanup​

When you're done experimenting, tear everything down:

anyscale workspace_v2 terminate --project default -n <workspace-name>

terraform destroy

Review the plan and type yes to confirm. This will remove all the resources, including the AKS cluster, Anyscale cloud registration, storage, and container registry.

Summary​

In this post, we walked through deploying Anyscale on Azure using Terraform with AKS managed Gateway API and Istio-based app routing. Along the way, we picked up a useful technique for figuring out API schemas when the docs haven't caught up yet, and saw how the azapi provider lets you automate new Azure services before the azurerm provider adds support.

The full Terraform code is available as a GitHub Gist. Clone it, tweak the variables, and you should be up and running in under 10 minutes.

What's next?​

• Explore the Anyscale on Azure documentation for more configuration options
• Check out the Ray documentation for building distributed applications
• Try scaling your Ray cluster with different node pool configurations and GPU-enabled VMs
• Look into Anyscale workspaces for interactive development on your cluster

Have questions or run into issues? Drop us a note in GitHub Discussions or open an issue.

Happy deploying!

Resources​

• Full Terraform code for this walkthrough
• Anyscale on Azure public preview announcement
• What is Anyscale on Azure?
• Quickstart: Deploy Anyscale on Azure
• AKS managed Gateway API
• Istio-based app routing with Gateway API on AKS
• AzAPI Terraform provider
• Ray documentation

Since our March preview announcement, we've also shipped two of our most requested capabilities that gated a real production story for Gateway API on AKS:

• Azure DNS and Azure Key Vault, wired in for you. No more manually deploying a SecretProviderClass, a sync pod, or a separate external-dns instance just to get a TLS-terminated, DNS-resolvable hostname. Drop two tls.options on a listener, apply an ExternalDNS CR, and you're done.
• Access logs, enabled out of the box. Every gateway proxy writes a structured JSON access log line per request to stdout. kubectl logs the gateway deployment and you're already debugging — no Telemetry resource, no EnvoyFilter, no opt-in flag. This closes one of the last parity gaps with the nginx experience our users are accustomed to.

The rest of this post walks through both. We'll create a cluster, expose a sample app over the Gateway API, see access logs land on stdout, and then layer on DNS + TLS using the ClusterExternalDNS and ExternalDNS CRDs plus listener TLS options. A working DNS zone and Key Vault are part of the demo, so by the end you'll have HTTPS records resolving in a zone you own.

What's new​

Access logs by default​

Every Envoy gateway proxy that AKS provisions for your Gateway resources now writes one structured JSON access log line per request to its container log out of the box. The very first thing you can do once traffic flows is kubectl logs the gateway deployment and watch requests arrive. No Istio resources needed to manage it, and no opt-in flag. This brings the Gateway API experience closer to parity with the nginx-based experience our users are accustomed to.

Because the logs land on stdout, they flow through the standard container log pipeline, so you can collect them centrally instead of tailing each gateway by hand. Enable the Azure Monitor (Container Insights) add-on, and each gateway proxy's access logs are collected into a Log Analytics workspace, where you can query, retain, and alert on them with KQL. You can enable it when you create or update the cluster (az aks create/update --enable-addons monitoring --workspace-resource-id <workspace-id>), or onboard an existing cluster through the portal — see Enable monitoring for Kubernetes clusters and the Monitor AKS overview. Once it's enabled, the per-request access log lines show up in the ContainerLogV2 table, so a query like ContainerLogV2 | where PodName startswith "httpbin-gateway-approuting-istio" gives you every request the gateway served across all replicas.

DNS and TLS integration via the App Routing operator​

At preview, DNS and TLS automation was a major piece of the ingress-nginx experience the Gateway API path didn't match. That gap is now closed via two paths in the App Routing operator:

• TLS — A pair of listener tls.options keys (kubernetes.azure.com/tls-cert-keyvault-uri and kubernetes.azure.com/tls-cert-service-account) tell the operator to provision a SecretProviderClass, sync the certificate from Azure Key Vault as a kubernetes.io/tls Secret, and patch the listener's certificateRefs for you.
• DNS — Two CRDs, ClusterExternalDNS (cluster-scoped) and ExternalDNS (namespace-scoped), let you stand up an external-dns instance scoped to one or more Azure DNS zones and tell it to watch Gateway/HTTPRoute/GRPCRoute resources. Records appear automatically based on the hostnames on your routes.

Demo: from cluster create to HTTPS-with-DNS​

The rest of this post is a hands-on walkthrough that takes you through our newly-GA'd features along with the enhancements we're so excited about. To follow along you'll need:

• Azure CLI 2.86.0 or later (az --version to check, az upgrade to update).
• An Azure subscription and resource group, plus permission to create resources in it.
• Sufficient Key Vault RBAC on your own Azure identity to create certificates — Key Vault Certificates Officer is the minimum, Key Vault Administrator is fine. Grant this on the vault you'll create in step 5b before running the cert-creation command.

Export these once and reuse them throughout:

export RESOURCE_GROUP=<your-resource-group>
export CLUSTER=<your-cluster-name>
export LOCATION=<your-azure-region>

1. Create the cluster (or update an existing one)​

The three flags that matter are --enable-app-routing-istio (turns on the App Routing Gateway API implementation), --enable-gateway-api (installs the AKS-managed Gateway API CRDs), and --enable-app-routing (deploys the App Routing operator, which manages the DNS/TLS integration). All APIs are GA — no preview-feature registration and no aks-preview extension required.

On a new cluster:

az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER \
  --location $LOCATION \
  --enable-app-routing-istio \
  --enable-app-routing \
  --enable-gateway-api \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --enable-addons azure-keyvault-secrets-provider

On an existing cluster:

az aks update \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER \
  --enable-app-routing \
  --enable-app-routing-istio \
  --enable-gateway-api \
  --enable-oidc-issuer \
  --enable-workload-identity

# CSI addon needs its own command on existing clusters
az aks enable-addons \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER \
  --addons azure-keyvault-secrets-provider

--enable-oidc-issuer and --enable-workload-identity are required for the DNS/TLS demo later — both DNS and TLS use Microsoft Entra Workload Identity to authenticate to Azure DNS and Key Vault. The azure-keyvault-secrets-provider addon installs the Secrets Store CSI Driver, which the operator uses to materialize Key Vault certificates as Kubernetes Secrets. Set them all now even if you don't have a workload to expose yet.

Pull credentials so kubectl works:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER

2. Confirm the control plane is up​

The App Routing Gateway API implementation runs istiod in the aks-istio-system namespace.

kubectl get pods -n aks-istio-system

NAME                      READY   STATUS    RESTARTS   AGE
istiod-54b4ff45cf-htph8   1/1     Running   0          3m15s
istiod-54b4ff45cf-wlvgd   1/1     Running   0          3m

You'll also see the app-routing-system namespace with NGINX still running:

kubectl get pods -n app-routing-system

You can ignore this for the rest of the demo. NGINX-by-default is on its way out as part of the Ingress-NGINX retirement. Once we stop deploying it after November 2026, it won't appear for users by default.

The approuting-istio GatewayClass is what you'll target with your Gateway resources:

kubectl get gatewayclass approuting-istio

NAME               CONTROLLER                    ACCEPTED   AGE
approuting-istio   istio.aks.azure.com/gateway-controller   True       4m

3. Expose a sample app via Gateway API​

We'll deploy httpbin, front it with a single Gateway listener over plain HTTP, attach an HTTPRoute, and verify traffic flows. (We'll make this HTTPS in section 5.)

Deploy httpbin in the default namespace:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/httpbin/httpbin.yaml

Create the Gateway and HTTPRoute:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  gatewayClassName: approuting-istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - name: httpbin-gateway
  hostnames: ["httpbin.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF

AKS provisions the Envoy Deployment, Service, HorizontalPodAutoscaler, and PodDisruptionBudget for the Gateway automatically. Wait for the Gateway to be programmed and grab its external IP:

kubectl wait --for=condition=programmed gateways.gateway.networking.k8s.io httpbin-gateway --timeout=120s
export INGRESS_HOST=$(kubectl get gateways.gateway.networking.k8s.io httpbin-gateway -ojsonpath='{.status.addresses[0].value}')

Send a request:

curl -s -I -H "Host: httpbin.example.com" "http://$INGRESS_HOST/get"

You should see an HTTP/1.1 200 OK.

4. See access logs on the gateway proxy​

Each Gateway resource is backed by a managed Deployment named <gateway-name>-approuting-istio:

kubectl get deployment httpbin-gateway-approuting-istio

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
httpbin-gateway-approuting-istio   2/2     2            2           1m

Tail its logs and re-issue a curl:

kubectl logs deploy/httpbin-gateway-approuting-istio --tail=20 -f

You should see an access log for each of your requests. That's the JSON-formatted Envoy access log written to stdout, on by default — useful for sanity checks, post-mortem analysis, or shipping into Azure Monitor / Log Analytics via the standard container log pipeline.

5. Add Azure DNS and TLS​

Now for the best part (well, at least in my opinion). We'll create an Azure DNS zone we own, an Azure Key Vault with a real cert for a hostname in that zone, a user-assigned managed identity that can write to both, and then wire everything in via two CRDs and two listener tls.options.

For this demo we'll use the zone ga-demo.dev with two sub-hosts, a.ga-demo.dev and b.ga-demo.dev. Substitute your own zone name.

export ZONE_NAME=ga-demo.dev
export KV_NAME=approuting-ga-demo-kv  # must be globally unique
export UAMI_NAME=approuting-ga-demo-id
export SA_NAME=approuting-demo-sa

5a. Create an Azure DNS zone​

az network dns zone create --resource-group $RESOURCE_GROUP --name $ZONE_NAME
export ZONE_ID=$(az network dns zone show --resource-group $RESOURCE_GROUP --name $ZONE_NAME --query id -o tsv)

The zone's authoritative nameservers are listed under az network dns zone show ... --query nameServers. You can find this on the Azure portal too. If you're using a registrar-owned domain in production, you'd point the registrar at those NS records. But for a one-off demo, you can resolve through the zone NS directly with dig @<ns> a.${ZONE_NAME} which is what we'll do later to ensure our curl resolves correctly.

5b. Create an Azure Key Vault and a self-signed wildcard cert​

We'll mint a self-signed wildcard cert directly in Key Vault so it covers both a.ga-demo.dev and b.ga-demo.dev from a single certificate. (For production, import a CA-issued PFX with az keyvault certificate import instead.)

az keyvault create \
  --name $KV_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --enable-rbac-authorization true

# Cert policy: wildcard subject + SAN so one cert covers every sub-host of the zone
cat > /tmp/cert-policy.json <<EOF
{
  "issuerParameters": { "name": "Self" },
  "keyProperties": { "exportable": true, "keyType": "RSA", "keySize": 2048, "reuseKey": false },
  "secretProperties": { "contentType": "application/x-pkcs12" },
  "x509CertificateProperties": {
    "subject": "CN=*.${ZONE_NAME}",
    "subjectAlternativeNames": { "dnsNames": ["*.${ZONE_NAME}", "${ZONE_NAME}"] },
    "validityInMonths": 12,
    "keyUsage": ["digitalSignature", "keyEncipherment"]
  }
}
EOF

az keyvault certificate create \
  --vault-name $KV_NAME \
  --name approuting-demo-cert \
  --policy @/tmp/cert-policy.json

# Capture the unversioned cert URI so the listener tracks rotations automatically
export CERT_URI=$(az keyvault certificate show \
  --vault-name $KV_NAME \
  --name approuting-demo-cert \
  --query id -o tsv | sed 's|/[^/]*$||')
echo "Cert URI: $CERT_URI"

5c. Create a managed identity, role assignments, and federated credentials​

Both the operator's DNS controller and the gateway listener's TLS sync authenticate to Azure as a user-assigned managed identity (UAMI), federated to a Kubernetes ServiceAccount via the cluster's OIDC issuer.

# 1. Create the UAMI
az identity create --resource-group $RESOURCE_GROUP --name $UAMI_NAME
export UAMI_CLIENT_ID=$(az identity show --resource-group $RESOURCE_GROUP --name $UAMI_NAME --query clientId -o tsv)
export UAMI_PRINCIPAL_ID=$(az identity show --resource-group $RESOURCE_GROUP --name $UAMI_NAME --query principalId -o tsv)

# 2. Grant DNS Zone Contributor on the zone, Key Vault Secrets User on the vault
az role assignment create \
  --assignee-object-id $UAMI_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "DNS Zone Contributor" \
  --scope $ZONE_ID

az role assignment create \
  --assignee-object-id $UAMI_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope $(az keyvault show --name $KV_NAME --query id -o tsv)

export OIDC_ISSUER=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv)

We'll create the app-a and app-b namespaces now and federate the same UAMI to a $SA_NAME ServiceAccount in each. Federated credentials are subject-specific — each (namespace, ServiceAccount) pair needs its own:

for ns in app-a app-b; do
  kubectl create namespace $ns

  az identity federated-credential create \
    --identity-name $UAMI_NAME \
    --resource-group $RESOURCE_GROUP \
    --name approuting-demo-fic-$ns \
    --issuer $OIDC_ISSUER \
    --subject "system:serviceaccount:$ns:$SA_NAME" \
    --audiences "api://AzureADTokenExchange"

  kubectl apply -n $ns -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: $SA_NAME
  annotations:
    azure.workload.identity/client-id: $UAMI_CLIENT_ID
  labels:
    azure.workload.identity/use: "true"
EOF
done

The azure.workload.identity/use: "true" label is required; it's what tells the AKS Workload Identity webhook to inject the projected OIDC token volume and the AZURE_* env vars into pods that consume this SA. Without it, the operator's placeholder pod gets a default Kubernetes SA token instead of one signed by your cluster's OIDC issuer, and Entra rejects it with AADSTS700211: No matching federated identity record.

The same ServiceAccount drives both the DNS controller and the listener's TLS sync. One identity, two roles in Azure, one SA per namespace.

5d. Deploy two gateways in two namespaces​

Deploy httpbin in each namespace, then create one Gateway per namespace with an HTTPS listener pointing at the Key Vault cert URI and the SA. Each gateway covers its own sub-host:

for ns in app-a app-b; do
  kubectl apply -n $ns -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/httpbin/httpbin.yaml
done

for pair in "app-a:a" "app-b:b"; do
  ns=${pair%%:*}; sub=${pair##*:}
  fqdn=${sub}.${ZONE_NAME}
  kubectl apply -n $ns -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: ${sub}-gateway
  labels:
    app: approuting-demo
    zone: ${sub}
spec:
  gatewayClassName: approuting-istio
  listeners:
  - name: https
    hostname: $fqdn
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        kubernetes.azure.com/tls-cert-keyvault-uri: $CERT_URI
        kubernetes.azure.com/tls-cert-service-account: $SA_NAME
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ${sub}-route
spec:
  parentRefs:
  - name: ${sub}-gateway
  hostnames: ["$fqdn"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF
done

The TLS options tell the App Routing operator to:

1. Resolve $CERT_URI against the Key Vault, using the workload identity bound to $SA_NAME in the listener's namespace.
2. Provision a SecretProviderClass named kv-gw-cert-<gateway>-<listener>.
3. Sync the certificate as a kubernetes.io/tls Secret of the same name in the listener's namespace.
4. Patch the listener's certificateRefs to point at the Secret.

You can confirm the SPCs and Secrets exist:

kubectl get secretproviderclass,secret -n app-a
kubectl get secretproviderclass,secret -n app-b

5e. Scenario A — ClusterExternalDNS: records for both gateways​

ClusterExternalDNS is cluster-scoped: a single CR points at one or more Azure DNS zones and watches Gateway resources cluster-wide. The operator deploys an external-dns instance in the namespace named under resourceNamespace; the SA in that namespace authenticates to Azure DNS.

Apply it:

kubectl apply -f - <<EOF
apiVersion: approuting.kubernetes.azure.com/v1alpha1
kind: ClusterExternalDNS
metadata:
  name: demo-cluster-dns
spec:
  resourceName: demo-cluster-dns
  resourceNamespace: app-a
  dnsZoneResourceIDs:
  - $ZONE_ID
  resourceTypes:
  - gateway
  identity:
    type: workloadIdentity
    serviceAccount: $SA_NAME
EOF

After a minute, both records show up in the zone:

az network dns record-set a list --resource-group $RESOURCE_GROUP --zone-name $ZONE_NAME -o table

Name    ResourceGroup              Ttl    Type    AutoRegistered    Metadata
------  -------------------------  -----  ------  ----------------  --------
a       <your-rg>                  300    A       False
b       <your-rg>                  300    A       False

That's ClusterExternalDNS: one CR, one external-dns deployment, records for both gateways across both namespaces.

5f. Scenario B — ExternalDNS: namespace-scoped, with label filtering​

Now flip the model. Delete the ClusterExternalDNS:

kubectl delete clusterexternaldns demo-cluster-dns

To make the namespace + label scope concrete, deploy a third gateway in app-a labeled zone=c. We'll point a namespace-scoped ExternalDNS at it and watch a fresh record appear — without touching the existing a and b records:

kubectl apply -n app-a -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: c-gateway
  labels:
    app: approuting-demo
    zone: c
spec:
  gatewayClassName: approuting-istio
  listeners:
  - name: https
    hostname: c.${ZONE_NAME}
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        kubernetes.azure.com/tls-cert-keyvault-uri: $CERT_URI
        kubernetes.azure.com/tls-cert-service-account: $SA_NAME
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: c-route
spec:
  parentRefs:
  - name: c-gateway
  hostnames: ["c.${ZONE_NAME}"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF
kubectl wait -n app-a --for=condition=programmed gateway c-gateway --timeout=240s

Now create a namespace-scoped ExternalDNS in app-a with a label filter for zone=c:

kubectl apply -n app-a -f - <<EOF
apiVersion: approuting.kubernetes.azure.com/v1alpha1
kind: ExternalDNS
metadata:
  name: demo-ns-dns
spec:
  resourceName: demo-ns-dns
  dnsZoneResourceIDs:
  - $ZONE_ID
  resourceTypes:
  - gateway
  identity:
    type: workloadIdentity
    serviceAccount: $SA_NAME
  filters:
    gatewayLabels: "zone=c"
EOF

Two scoping rules apply at once:

1. Namespace scope — ExternalDNS only watches resources in its own namespace, so b-gateway in app-b is invisible to it.
2. Label filter — within app-a, only gateways carrying zone=c are picked up. The label key/value is rendered into the underlying external-dns controller's label selector.

After about a minute, a new c record appears alongside whatever was already there from the cluster-scoped run:

az network dns record-set a list --resource-group $RESOURCE_GROUP --zone-name $ZONE_NAME -o table

Name    ResourceGroup              Ttl    Type    AutoRegistered    Metadata
------  -------------------------  -----  ------  ----------------  --------
a       <your-rg>                  300    A       False
b       <your-rg>                  300    A       False
c       <your-rg>                  300    A       False

The new namespace-scoped controller is responsible for c only. a-gateway (also in app-a) is excluded by the label filter; b-gateway is excluded by the namespace filter.

5g. Verify HTTPS end-to-end​

Grab the zone's authoritative nameserver, resolve the FQDN through it, and feed the result straight into curl --resolve. That keeps the test self-contained — you don't need the registrar pointing at Azure DNS or your local resolver to know about the zone:

NS=$(az network dns zone show --resource-group $RESOURCE_GROUP --name $ZONE_NAME --query 'nameServers[0]' -o tsv | sed 's/\.$//')
GATEWAY_IP=$(dig +short @${NS} a.${ZONE_NAME} | tail -1)
echo "Resolved a.${ZONE_NAME} -> $GATEWAY_IP via $NS"

# Self-signed cert, so -k. Use --cacert if you exported the chain.
curl -k -I --resolve "a.${ZONE_NAME}:443:${GATEWAY_IP}" "https://a.${ZONE_NAME}/get"

You should see HTTP/2 200. The certificate Envoy presents is the one synced from Key Vault. You can re-tail the gateway deployment's logs (kubectl logs deploy/a-gateway-approuting-istio -n app-a) to see the access log line for the request.

Next steps​

• Migrate: If you're on the NGINX add-on, see the Ingress to Gateway API migration guide. Managed NGINX support continues through November 2026.
• Deepen: The full reference docs are at Configure ingress with the Kubernetes Gateway API and Secure ingress with the application routing Gateway API implementation.
• Track: AKS release tracker and the AKS release notes.
• Feedback: Open an issue at Azure/AKS issues.

Our vision for Azure Kubernetes Service has always been simple: AKS runs everywhere. The same Azure Kubernetes Service you run across Azure regions is the same platform you run in sovereign and private clouds, and on OEM hardware. One platform, every substrate — so that the skills, the tooling, and the operational model you already know come with you wherever your applications need to run.

Today, we're adding the newest member to that family — and it answers one of the strongest signals we've heard from customers at the edge. AKS now runs directly on bare-metal, small-form-factor devices at the edge, available today in public preview.

Why bare metal​

For years, customers have told us the same thing: the edge forces hard trade-offs on Kubernetes. They want the consistency of a managed Kubernetes platform in the places where their business actually happens — the retail back room, the factory floor, the quick-service restaurant, the branch office, the remote site. But the edge is not the datacenter. Space, power, and budget are constrained. Hardware is small and sometimes ruggedized. Connectivity is intermittent, and sites often need to keep running through disconnection. Given the Kubernetes control plane runs locally on the device, deployed workloads continue to operate normally during connectivity loss. Only portal visibility and Azure management actions are interrupted until the connection is restored.

What customers have asked us for is precise: run real Kubernetes directly on the hardware, with no hypervisor underneath and no virtualization layer to license or maintain; do it on lightweight edge hardware rather than datacenter-class servers; and let it keep operating through intermittent connectivity.

That is exactly what AKS on bare metal delivers. This is Azure Kubernetes Service running directly on the device with no hypervisor, on small-form-factor hardware, and resilient to intermittent connectivity. It is not a different product for the edge. It is the same AKS, supporting a new class of hardware at the edge.

From power-on to a running AKS cluster​

The experience is deliberately simple. On-site, someone plugs in a USB drive with an OS image from the Azure portal, powers on the device, waits a couple of minutes, and removes the drive. That's the only step that happens at the location. During this zero-touch provisioning process, a device voucher is written back to the USB drive and all configuration shifts to Azure — no deep infrastructure expertise required on site.

From there, everything happens in Azure. The machine is onboarded through the Azure Local small form factor creation experience — create a site, upload the voucher, and the machine comes up as Provisioned. The cluster creation looks exactly like AKS anywhere else: configure Azure RBAC, set the control-plane networking, enable Azure Monitor, and deploy. When it completes, you have an AKS cluster running on Azure Linux, on bare metal, fully manageable from Azure: workloads, services, ingress, and Kubernetes upgrades, all without leaving the portal.

One consistent AKS experience, from cloud to edge​

Provisioning is only half the story. What matters at the edge is how you operate the cluster over time — and here the principle is simple: AKS on bare metal is just an AKS cluster. There's no edge-specific console, no separate operational model, no exceptions to learn.

Because it's a first-class AKS cluster, it shows up in Azure Kubernetes Fleet Manager right alongside your cloud clusters, managed and monitored with the same controls and the same tooling you already use everywhere else. The same AKS, the same experience, consistent across cloud and edge — now including bare metal.

Available today​

This is what "AKS everywhere" was always meant to become: not a fragmented collection of edge-specific products and tools, but one Kubernetes platform that follows your applications to every place they need to run — now including the smallest, most constrained, and most remote hardware at the edge.

AKS on bare metal is now in public preview. We can't wait to see what you build on it.

Join the preview today at aka.ms/aks-edge-baremetal.

In the first three blog posts of this series, we introduced NVIDIA Dynamo on AKS, covered SLO-driven scaling with the Dynamo Planner and Profiler, and explored KV-cache-aware routing to establish the foundations for high-performance LLM serving. In this post, we move up one layer of the inference stack: how to describe and operate the distributed inference workload on a Kubernetes cluster.

As inference deployments evolve from simple model-serving replicas to systems with disaggregated prefill and decode phases, multi-node model instances, and explicit startup dependencies, an inference deployment benefits from a single API that describes the full serving pipeline: which roles exist, how they relate, and what constitutes a deployable unit of inference-serving capacity.

NVIDIA Grove addresses this layer of the stack with a Kubernetes-native API for describing an AI inference service — so the platform can schedule, start, and scale the deployment in units that match the inference architecture.

The Kubernetes orchestration gap for distributed inference​

Modern AI inference workloads are fundamentally different from the traditional distributed workloads built with independent, stateless microservices. A production LLM deployment may include distinct roles – frontend services, KV-aware routers, prefill and decode workers – that are tightly coupled in how they must be scheduled, started, and scaled. They require scheduling in groups, starting up in a specific order, and scaling in ratios that preserve an end-to-end serving path. For larger models, a single prefill or decode instance may span several pods; where a complete deployment of each instance is necessary for a functional serving capacity.

Kubernetes primitives can independently schedule and run these components, but they do not natively understand the inter-component relationships required for the deployment to operate as an inference service:

• Which pods form one usable model instance?
• Which roles must start before others?
• Which components scale independently, and which must scale as a group?

Using existing Kubernetes primitives alone, operations teams are forced to spread these considerations across bespoke manifests, labels, scripts, or custom controllers. This can create orchestration challenges and worsen the deployment observability.

LeaderWorkerSet (LWS) is an Upstream API that addresses a part of this problem by managing a leader pod and its worker pods as a replicated unit, which generally maps well to multi-node model instances. A full Dynamo-style deployment, however, can contain multiple leader-worker units across prefill and decode, plus routing and other service components. LWS may model those leader-worker groups, but it is not intended to describe the full inference service across them. Additionally, LWS couples multi-node scaling units with a leader-worker pattern, and not all multi-node use cases map to this cleanly. Overall, scheduling, startup, and scaling for such modern deployments require a more flexible Kubernetes-native workload abstraction.

NVIDIA Grove for Kubernetes-native AI scheduling​

NVIDIA Grove is an open-source Kubernetes API, available as a modular component of NVIDIA Dynamo, for defining both simple single-node and complex multi-node AI workloads. Where standard Kubernetes resources describe individual pods and services, Grove enables you to describe an entire inference serving system’s components and orchestration requirements as one object: routing, prefill, decode, leader-worker groups/multi-node units, startup dependencies, and scaling boundaries — all expressed in a single workload specification.

From that specification, Grove takes responsibility for coordinating the Kubernetes resources required to run the service. Scheduling, startup, and scaling have a shared view of the workload: which pods form a usable instance and which roles belong together. Scaling actions can target a single role, a complete multi-node model instance, or a full service replica. Startup dependencies between roles are part of the workload definition rather than external scripts.

The result is a workload API that matches how distributed inference systems are actually built: individual roles with different resource profiles, grouped model instances that must scale together, and complete service replicas that represent usable end-to-end serving capacity.

In an inference deployment, Dynamo provides the serving stack: inference workers, routing, the Planner, KV cache management, and backend integrations such as vLLM, TensorRT-LLM, and SGLang. Grove defines the Kubernetes orchestration layer around this stack, a.k.a. the contract between the inference system and the cluster. Grove can also be deployed standalone or integrated with other high-performance inference frameworks.

How Grove models the workload​

Grove's user-facing model is built around three hierarchical primitives that together describe the full shape of an inference deployment, from individual roles up to the complete service:

Using this hierarchy, Grove generates the lower-level Kubernetes resources the cluster needs to schedule, start, and scale the workload. Users work at the inference deployment level, while Grove translates the intent into Grove managed pods and other resources, including the scheduler-facing PodGang resources that encode the scheduling constraints.

Coordinating scheduling, startup, and scalability​

Once Grove has the workload specification, it uses the same hierarchy to guide how the workload is scheduled, started, and scaled.

Scheduling​

For multi-node inference, serving capacity arrives as a collection of complete instances. A prefill or decode instance that spans several pods can serve traffic only when the full set can run together. Gang scheduling gives Kubernetes an all-or-nothing contract: either all pods in the instance are scheduled together, or none are. Grove creates the scheduler-facing PodGang resources automatically from the workload specification, so this guarantee is part of the workload model rather than a rule that teams have to enforce separately.

Gang scheduling ensures that each individual component of a deployment is allocated as a complete unit, for example, a full prefill instance or a full decode instance. However, this alone does not guarantee that the deployment can actually serve end-to-end requests: the scheduler might schedule several complete prefill instances without any matching decode capacity, or vice versa.

Grove addresses this limitation with hierarchical gang scheduling. Instead of enforcing completeness only at the instance level, Grove also enforces it at the service level. The workload specifies higher-level viability constraints, for example, that at least one complete prefill instance and one complete decode instance must be scheduled together before the deployment is considered ready. The scheduler therefore treats these interdependent components as a “higher-level gang”: it schedules the required combination only when enough resources are available. This prevents the system from allocating individually complete but functionally unusable configurations that cannot process requests end to end.

On an AKS cluster, Grove-managed inference pods can be scheduled by a gang-aware scheduler like the KAI Scheduler, an open-source Kubernetes-native scheduler built for AI workloads with native PodGang awareness. Grove defines the inference workload - the roles, groups, dependencies, and scaling boundaries - while KAI gives Kubernetes the scheduling behavior needed to place those pods as complete units. Together, they enable these capabilities to be expressed and enforced cleanly within the Kubernetes-native infrastructure.

Startup​

Scheduling is only the first step. Distributed serving systems often have startup dependencies; for example, workers need to be discoverable before leaders start, and routers or frontends may need to be ready before model workers register. Grove integrates startup order with the workload definition through cliqueStartupType and startsAfter, instead of leaving it to scripts or external conventions.

Scaling​

Grove has multiple scaling boundaries in a hierarchy model: a PodClique for one role, a PodCliqueScalingGroup for a complete leader-worker unit, and a PodCliqueSet for the full service. This preserves independent scaling even when scheduling is coordinated: prefill and decode can be scheduled as viable units while still scaling separately in response to various bottlenecks.

Different scaling boundaries also integrate well with the Dynamo Planner (covered in Part 2 of this blog series). While the Planner reasons about traffic shape, sequence lengths, and time-to-first-token (TTFT) or inter-token latency (ITL) targets to decide when capacity should change, Grove ensures this decision targets the appropriate Grove workload boundary: a PodClique for a single role, a PodCliqueScalingGroup for a complete leader-worker instance, or the PodCliqueSet for the end-to-end inference service. Ultimately, Planner-driven scale-out doesn't just add pods; it adds a complete, gang-scheduled inference instance that is ready to serve.

Modeling inference deployments with Grove​

Grove's PodCliqueSet API is flexible enough to cover a range of inference deployment shapes:

Aggregated serving: where a single PodClique is sufficient; each worker handles the full request lifecycle, and scaling means adding more copies of that worker pod.

Disaggregated serving: router, prefill, and decode become separate PodCliques, each scaling independently in response to different traffic pressures.

Multi-node: when a single model instance spans multiple pods, a PodCliqueScalingGroup binds the leader and worker PodCliques into one scaling unit.

The Grove GitHub repository includes reference manifests for these deployment patterns, giving teams concrete starting points for mapping their own serving architecture into PodCliques, PodCliqueScalingGroups, and PodCliqueSets.

See Grove in action below as it orchestrates a multi-node Llama 3.1 70B deployment on AKS. The demo serves Llama-3.1-70B-Instruct-FP8 on 16x NVIDIA H100 (2 nodes of Azure Standard_ND96isr_H100_v5) with tp=16. The walkthrough shows a DynamoGraphDeployment being translated into Grove resources, worker pods landing across H100 nodes, and scale-out adding a second gang-scheduled two-pod worker as a grouped unit.

Looking ahead​

By introducing this workload layer, Grove provides Kubernetes with a structured contract for orchestrating the gang scheduling and scaling requirements of AI workloads. This also provides the foundation for handling complex challenges that modern inference workloads are facing today.

One such challenge is topology-aware scheduling. As deployments span multiple nodes and accelerator domains, scheduling decisions become critical to performance. Without topology awareness, gang-scheduled instances may be placed on nodes that are physically or network-topologically distant from one another, introducing communication bottlenecks that undermine the efficiency that the inference stack is designed to deliver.

Rolling updates introduce another layer of complexity. In disaggregated inference, prefill and decode components usually have compatibility constraints across framework versions, requiring cross-version traffic to be blocked during upgrades. This can strand capacity across versions: for example, a rollout might leave eight old-version prefill instances but only two old-version decode instances, reducing the amount of end-to-end capacity that can actually serve requests. Updating these systems safely requires rollout behavior that preserves viable serving capacity across the pipeline, rather than updating each role as an independent pool of pods.

In Part 5, we’ll build on the Grove workload model introduced here and explore how this structured view of the serving pipeline enables more advanced orchestration, including topology-aware scheduling for distributed inference and safer rolling updates for disaggregated serving pipelines. We’ll also share how we’re collaborating with the NVIDIA Grove and Dynamo communities toward a shared goal: building a Kubernetes-native orchestration layer for the demands of modern inference.

The approaches covered in this post apply mitigations as follows:

1. Immediate mitigation: deploy a DaemonSet using Fleet Manager resource placement. Best for clusters where a node image upgrade can't yet be applied.
2. Permanent mitigation: upgrade node images using Fleet Manager update runs. Applies patches when new node image version is available in the cluster's region.

The original AKS advisory and detailed mitigation guide for per-cluster application can be found in GitHub Issue 5753.

Before you begin​

• Azure CLI version 2.86.0 or later installed (instructions).
• Fleet Manager Azure CLI extension installed and updated.

az extension add --name fleet
az extension update --name fleet

• A Fleet Manager with a hub cluster (instructions). You can add a hub cluster if you are already using Fleet without one.
• AKS clusters added as member clusters with update group and member labels set. Groups and labels are required for controlling the rollout via placement and update runs.

az fleet member create \
  --resource-group $GROUP \
  --fleet-name $FLEET \
  --name $MEMBER_CLUSTER_NAME \
  --update-group canary \
  --labels upgroup=canary

• kubectl configured for the hub cluster (instructions).

Immediate mitigation - DaemonSet rollout​

For immediate protection, apply a DaemonSet that disables the vulnerable kernel module. The DaemonSet is deployed into its own Namespace on each cluster, making it easy to discern intent and easier to distribute across clusters. Given how DaemonSets function, the protection is applied to the node, even though we don't use the kube-system Namespace as shown in the original AKS mitigation guide.

Fleet Manager's cluster-scoped resource placement enables:

• Multi-cluster selection for a resource via ClusterResourcePlacement.
• Controlled rollout of the resource via ClusterStagedUpdateRun.

Step 1: Create the mitigation Namespace and DaemonSet​

Create the 01-kernel-lpe-mitigate.yaml file with the following contents. The DaemonSet blocks vulnerable modules covered by the CVEs.

apiVersion: v1
kind: Namespace
metadata:
  name: kernel-lpe-cve-mitigate-ns
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kernel-lpe-mitigate
  namespace: kernel-lpe-cve-mitigate-ns
  labels:
    app: kernel-lpe-mitigate
    purpose: security-mitigation
spec:
  selector:
    matchLabels:
      app: kernel-lpe-mitigate
  template:
    metadata:
      labels:
        app: kernel-lpe-mitigate
    spec:
      hostPID: true
      priorityClassName: system-node-critical
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - operator: Exists
      containers:
        - name: mitigate
          image: mcr.microsoft.com/cbl-mariner/busybox:2.0
          command:
            - /bin/sh
            - -c
            - |
              echo "=== Kernel LPE Module Mitigation ==="
              echo "Covers: CVE-2026-31431 (algif_aead), DirtyFrag (esp4/esp6/rxrpc)"

              MODULES="algif_aead esp4 esp6 rxrpc"

              for mod in $MODULES; do
                if ! grep -qs "install ${mod} /bin/false" /host/etc/modprobe.d/*.conf 2>/dev/null; then
                  printf "install %s /bin/false\nblacklist %s\n" "$mod" "$mod" >> /host/etc/modprobe.d/disable-kernel-lpe.conf
                  echo "Blocked ${mod}"
                else
                  echo "${mod} already blocked"
                fi

                if chroot /host grep -q "^${mod} " /proc/modules 2>/dev/null; then
                  if chroot /host modprobe -r "$mod" 2>/dev/null; then
                    echo "Unloaded ${mod}"
                  else
                    echo "WARNING: Could not unload ${mod} (in use). Reboot node."
                  fi
                fi
              done

              echo "=== Mitigation complete. Sleeping ==="
              sleep infinity
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 32Mi
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-root
              mountPath: /host
      volumes:
        - name: host-root
          hostPath:
            path: /
            type: Directory

Stage the mitigation namespace and DaemonSet on the Fleet Manager hub cluster.

kubectl apply -f 01-kernel-lpe-mitigate.yaml

Step 2: Create a ClusterResourcePlacement​

Create a placement manifest 02-fleet-pick-clusters-mitigation.yaml that selects all Fleet Manager member clusters (PickAll) as suitable to receive the Namespace and its DaemonSet. We set the strategy to External so we can define and control the rollout order for clusters, which we will do in following steps.

apiVersion: placement.kubernetes-fleet.io/v1
kind: ClusterResourcePlacement
metadata:
  name: kernel-lpe-cve-mit-ns-place
spec:
  resourceSelectors:
  - group: ""
    version: v1
    kind: Namespace
    name: kernel-lpe-cve-mitigate-ns
  policy:
    placementType: PickAll
  strategy:
    type: External

Apply the placement on Fleet Manager hub cluster.

kubectl apply -f 02-fleet-pick-clusters-mitigation.yaml

You can validate how many clusters will receive the mitigation using this command.

kubectl get clusterresourceplacement kernel-lpe-cve-mit-ns-place -o jsonpath="{.status.conditions[?(@.type=='ClusterResourcePlacementScheduled')].message}"

The message in the response shows how many clusters were selected.

found all cluster needed as specified by the scheduling policy, found 2 cluster(s)

Step 3: Create a ClusterStagedUpdateStrategy​

A strategy is used to define the rollout order and soak duration per rollout stage. Note that all selected clusters must be in a stage. This is determined by the labels set of the cluster when it was added to the Fleet Manager.

In this sample 03-fleet-mitigation-rollout-strategy.yaml we have three stages, with a 4 hour soak time between each stage. You can also add approvals if required.

apiVersion: placement.kubernetes-fleet.io/v1
kind: ClusterStagedUpdateStrategy
metadata:
  name: kernel-lpe-cve-mit-ds-strategy
spec:
  stages:
    - name: stage1
      labelSelector:
        matchLabels:
          upgroup: canary
      afterStageTasks:
        - type: TimedWait
          waitTime: 4h
      maxConcurrency: 1
    - name: stage2
      labelSelector:
        matchLabels:
          upgroup: nonprod
      afterStageTasks:
        - type: TimedWait
          waitTime: 4h
      maxConcurrency: 1
    - name: stage3
      labelSelector:
        matchLabels:
          upgroup: prod

Apply the strategy on Fleet Manager hub cluster.

kubectl apply -f 03-fleet-mitigation-rollout-strategy.yaml

Step 4: Start rollout​

Finally, create a ClusterStagedUpdateRun 04-fleet-mitigation-rollout-run.yaml to begin the rollout.

apiVersion: placement.kubernetes-fleet.io/v1
kind: ClusterStagedUpdateRun
metadata:
  name: kernel-lpe-cve-mit-ds-rollout
spec:
  placementName: kernel-lpe-cve-mit-ns-place
  stagedRolloutStrategyName: kernel-lpe-cve-mit-ds-strategy
  state: Run

Apply the update run on Fleet Manager hub cluster. If you have Run as the state, the rollout begins.

kubectl apply -f 04-fleet-mitigation-rollout-run.yaml

Step 5: Monitor and control rollout​

Use the following command to retrieve the status of the rollout. Inspect the conditions and stageStatuses arrays to review progress.

kubectl get clusterstagedupdaterun kernel-lpe-cve-mit-ds-rollout -o yaml

If you want to stop the rollout, you can patch the update run:

kubectl patch clusterstagedupdaterun kernel-lpe-cve-mit-ds-rollout --type merge -p '{"spec":{"state":"Stop"}}'

Restart the rollout by patching the update run:

kubectl patch clusterstagedupdaterun kernel-lpe-cve-mit-ds-rollout --type merge -p '{"spec":{"state":"Run"}}'

Step 6: Validate mitigation is applied​

Select clusters in your fleet that have received the mitigation and run the following check.

kubectl logs -n kernel-lpe-cve-mitigate-ns -l app=kernel-lpe-mitigate

If the mitigation is applied successfully you should see.

=== Kernel LPE Module Mitigation ===
Covers: CVE-2026-31431 (algif_aead), DirtyFrag (esp4/esp6/rxrpc)
Blocked algif_aead
Blocked esp4 
Blocked esp6
Blocked rxrpc
=== Mitigation complete. Sleeping ===

Step 7: Roll back or remove​

If issues occur, or the DaemonSet is no longer required as the node image has been updated to a permanently patched release, you can remove the mitigation DaemonSet by deleting the ClusterResourcePlacement on the Fleet Manager hub cluster.

kubectl delete clusterresourceplacement kernel-lpe-cve-mit-ns-place

DaemonSet Rollout Outcome​

• Immediate mitigation applied across clusters via DaemonSet.
• Controlled rollout reduces risk, allows testing on clusters in batches.
• No node reboot required when module not in use.

Permanent mitigation - node image upgrade​

Once AKS provides a permanent mitigation via updated node images you can use Fleet Manager update runs to update the node image across your clusters.

Fleet Manager's safe multi-cluster updates feature enables:

• Application of consistent node image across clusters using the Node Image channel.
• Controlled rollout of the node image update via Update Strategies.

Step 1: Define an update strategy​

An update strategy defines the order in which clusters will receive the node image update. Strategies are defined using a simple JSON format as shown next.

In the following sample we have one group of clusters per stage, with clusters grouped based on the update-group on the member cluster. We continue with the 4 hour soak time and have a 100% concurrency which allows up to 50 clusters to update at once. You can also set these up via the Azure portal if desired.

{
    "stages": [
        {
            "name": "stage-1",
            "maxConcurrency": "100%",
            "groups": [
                {
                    "name": "canary",
                    "maxConcurrency": "100%"
                }
            ],
            "afterStageWaitInSeconds": 14400
        },
        {
            "name": "stage-2",
            "maxConcurrency": "100%",
            "groups": [
                {
                    "name": "nonprod",
                    "maxConcurrency": "100%"
                }
            ],
            "afterStageWaitInSeconds": 14400
        },
        {
            "name": "stage-3",
            "maxConcurrency": "100%",
            "groups": [
                {
                    "name": "prod",
                    "maxConcurrency": "100%"
                }
            ]
        }
    ]
}

Save the strategy as a file (nodeimage-strategy.json) and then submit via the Azure CLI to create the Strategy ready for use in Fleet Manager.

export GROUP=resource-group
export FLEET=fleet-name
export STRATEGY=copyfail-mitigation-strategy

az fleet updatestrategy create \
  --resource-group $GROUP \
  --fleet-name $FLEET \
  --name $STRATEGY \
  --stages nodeimage-strategy.json

Step 2: Create an update run for node image upgrades​

Next, let's create a new update run for updating just the node image on selected clusters.

The update run won't automatically start, so you can create this once AKS releases the patched node images, then start the run at a later time that suits your schedule.

az fleet updaterun create \
  --resource-group $GROUP \
  --fleet-name $FLEET \
  --name copyfail-nodeimage-update \
  --upgrade-type NodeImageOnly \
  --update-strategy-name $STRATEGY

Step 3: Start the update run​

Once you are satisfied that all clusters to be upgraded are included, you can start the update run.

az fleet updaterun start \
  --fleet-name $FLEET \
  --resource-group $GROUP \
  --name copyfail-nodeimage-update

Step 4: Monitor progress​

You can review the status using the Azure CLI shown below, or use the Azure portal's web interface to view the rollout graphically.

az fleet updaterun show \
  --resource-group $GROUP \
  --fleet-name $FLEET \
  --name copyfail-nodeimage-update

Node Image Upgrade Outcome​

• All nodes are updated to the patched node image.
• DaemonSet no longer required and can be removed.

Summary and next steps​

In this post we've looked at the multiple ways in which Azure Kubernetes Fleet Manager can help you safely apply mitigations for the Copy Fail CVE across all your clusters.

Fleet Manager continues to add new capabilities regularly, with the goal of simplifying at-scale Kubernetes cluster management for everyone.

Here's some recommended next steps:

• Review AKS Advisory and Mitigation for CVE-2026-31431 (Copy Fail).
• Learn more about Fleet Manager Safe Multi-cluster Updates.
• Learn more about Fleet Manager resource placement.
• View the Fleet Manager public roadmap. Feature requests always welcome!
• Join a KubeFleet community call to learn about Fleet Manager's open source resource placement.

In this post, we'll focus on the identity side of that story. We'll walk through integrating Argo CD with Microsoft Entra ID to give your developers SSO access using their existing corporate identities; no separate passwords to manage. You also get the full power of Azure's identity platform: multi-factor authentication (MFA), conditional access policies, and group-based role assignments to control who can do what in Argo CD. We'll automate the entire setup with Terraform so the configuration is repeatable, version-controlled, and easy to adapt across environments.

Overview​

By now, Argo CD probably needs no introduction. But if you are new to Argo CD, it is a declarative, GitOps continuous delivery tool for Kubernetes. It allows you to manage your Kubernetes applications using Git repositories as the source of truth.

Argo CD is heavily used in production environments and many have already adopted the open-source version of Argo CD. With the new Argo CD extension for AKS, you can now easily deploy the software on your cluster as an Azure-native resource. But what does that mean? Well, for one, it means you can now leverage Azure CLI, Azure Resource Manager (ARM) templates, Bicep, or Terraform to manage your Argo CD deployments as you bootstrap clusters. It also means you no longer need to worry about upgrading and patching Argo CD. Azure handles that for you, so you can focus on deploying applications.

By default, the Argo CD installation comes with a built-in admin user. However, for production environments, it is recommended to use an external identity provider for authentication and authorization. Microsoft Entra ID is a great choice for this as it provides robust security features and seamless integration with Azure services. My motto has always been "ditch the passwords" and using Microsoft Entra ID for authentication allows you to do just that. You can leverage features like multi-factor authentication (MFA), conditional access policies, and role-based access control (RBAC) to secure your Argo CD deployment.

At a high level, the process of securing Argo CD with Microsoft Entra ID involves the following steps:

1. Create a Microsoft Entra ID application for Argo CD
2. Create OIDC configuration settings to connect Microsoft Entra ID and enable workload identity for Argo CD
3. Create policy mappings to grant Microsoft Entra ID users/groups access to Argo CD
4. Create an AKS cluster with OIDC issuer and workload identity enabled
5. Deploy the Argo CD extension for AKS
6. Verify the integration

There's a bit of prerequisite setup to go through, especially when it comes to configuring Microsoft Entra ID applications and OIDC settings, but don't worry, we'll automate all of that using Terraform and I'll go through each step in detail. The end result will be a secure Argo CD deployment that is integrated with Microsoft Entra ID for authentication and authorization.

Step 0: Prerequisites​

Before you get started, make sure you have the following prerequisites in place:

• Azure subscription
• Azure CLI
• kubectl
• Terraform
• Argo CD CLI (optional)

Within your Azure tenant/subscription, you will also need to have the necessary permissions to create Microsoft Entra ID applications and manage Azure resources.

Unless you are the Global Administrator in your tenant, you will need to have one of the following roles to be able to create Microsoft Entra ID applications:

• Application Administrator
• Application Developer
• Cloud Application Administrator

Step 1: Terraform provider setup​

Create a new directory for your Terraform configuration and navigate to it and create a new file named main.tf.

In this file, we will define the Terraform providers for Azure Resource Manager (azurerm) and Microsoft Entra ID (azuread). The azurerm provider will allow us to manage Azure resources, while the azuread provider will allow us to manage Microsoft Entra ID applications and related resources.

terraform {
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.8.0"
    }

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.69.0"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

Step 2: Create Microsoft Entra ID application for Argo CD​

Next, let's create some data sources to query Microsoft Entra ID for the necessary information to set up our application and OIDC configuration. Add the following code to your main.tf file to get a list of well-known application IDs, the current client's configuration, and the service principal for Microsoft Graph.

data "azuread_application_published_app_ids" "well_known" {}
data "azuread_client_config" "current" {}
data "azuread_service_principal" "msgraph" {
  client_id = data.azuread_application_published_app_ids.well_known.result["MicrosoftGraph"]
}

Since the OIDC configuration for Argo CD requires the IDs of specific Microsoft Graph scopes, we will also create a local variable named msgraph_scopes that includes all the scopes from the Microsoft Graph service principal. This will allow us to easily reference the scope IDs when we create our Microsoft Entra ID application for Argo CD.

Add the following code to your main.tf file to create the msgraph_scopes local variable:

locals {
  msgraph_scopes = { for s in data.azuread_service_principal.msgraph.oauth2_permission_scopes : s.value => s }
}

Now we are ready to create our Microsoft Entra ID application for Argo CD. This application will represent your Argo CD in Microsoft Entra ID and will be used for authentication and authorization.

Add the following code to your main.tf file to create the application:

resource "azuread_application" "example" {
  display_name            = "myargocdapp"
  owners                  = [data.azuread_client_config.current.object_id]
  sign_in_audience        = "AzureADMyOrg"
  group_membership_claims = ["ApplicationGroup"]

  web {
    redirect_uris = [
      "https://localhost:9000/auth/callback"
    ]
  }

  public_client {
    redirect_uris = [
      "http://localhost:8085/auth/callback"
    ]
  }

  required_resource_access {
    resource_app_id = "00000003-0000-0000-c000-000000000000" # Microsoft Graph

    resource_access {
      id   = local.msgraph_scopes["openid"].id
      type = "Scope"
    }

    resource_access {
      id   = local.msgraph_scopes["profile"].id
      type = "Scope"
    }

    resource_access {
      id   = local.msgraph_scopes["email"].id
      type = "Scope"
    }

    resource_access {
      id   = local.msgraph_scopes["User.Read"].id
      type = "Scope"
    }
  }

  optional_claims {
    id_token {
      name      = "groups"
      essential = true
    }
  }
}

Here's what each part of the application configuration does:

• sign_in_audience = "AzureADMyOrg" — restricts sign-in to users in your Microsoft Entra ID tenant only
• group_membership_claims = ["ApplicationGroup"] — includes group membership in the token, which we'll use later for RBAC role mapping
• web and public_client redirect URIs — required for the OIDC authentication flow with Argo CD (the web URI is for the browser-based flow, and the public_client URI is for the CLI). Since we're testing locally, we use localhost URIs here, but for production you'll want to update these to match your deployment
• required_resource_access — requests Microsoft Graph API permissions (openid, profile, email, and User.Read) needed for the OIDC flow and to retrieve user information
• optional_claims — includes the user's group memberships in the ID token, enabling group-based access control in Argo CD

The workload identity integration requires Argo CD to exchange its Kubernetes service account token for a Microsoft Entra ID token, so we will need to create a federated identity credential for the application to establish trust between Microsoft Entra ID and the cluster's OIDC issuer.

Add the following code to your main.tf file to create the federated identity credential for the application:

resource "azuread_application_federated_identity_credential" "example" {
  application_id = azuread_application.example.id
  display_name   = azuread_application.example.display_name
  audiences      = ["api://AzureADTokenExchange"]
  issuer         = azurerm_kubernetes_cluster.example.oidc_issuer_url
  subject        = "system:serviceaccount:argocd:argocd-server"
}

Next, we need to create a service principal for the application. The service principal is what will be used to grant permissions and access to resources in Azure.

resource "azuread_service_principal" "example" {
  client_id = azuread_application.example.client_id
}

Now we need to grant the service principal the necessary permissions to access Microsoft Graph on behalf of the user. Add the following code to your main.tf file to create the delegated permission grant:

resource "azuread_service_principal_delegated_permission_grant" "example" {
  service_principal_object_id          = azuread_service_principal.example.object_id
  resource_service_principal_object_id = data.azuread_service_principal.msgraph.object_id
  claim_values                         = ["openid", "profile", "email", "User.Read"]
}

With the service principal and permissions in place, we can now grant Microsoft Entra ID users and groups access to Argo CD by creating app role assignments. Add the following code to your main.tf file to retrieve a Microsoft Entra ID group object ID.

data "azuread_group" "example" {
  display_name = "Argo CD Admins" # Update this to match your group name
}

Finally, add the following code to your main.tf file to allow members of the Microsoft Entra ID group to have access to the Argo CD application.

resource "azuread_app_role_assignment" "example" {
  app_role_id         = "00000000-0000-0000-0000-000000000000" # Default app role
  principal_object_id = data.azuread_group.example.object_id
  resource_object_id  = azuread_service_principal.example.object_id
}

Step 3: Create AKS cluster​

Now comes the easy part, creating the AKS cluster 😎

Add the following code to your main.tf file to create a resource group:

resource "azurerm_resource_group" "example" {
  name     = "myresourcegroup"
  location = "westus3" # Update to your preferred region
}

Add the following code to your main.tf file to create an AKS cluster with OIDC issuer and workload identity enabled:

resource "azurerm_kubernetes_cluster" "example" {
  name                      = "myakscluster"
  location                  = azurerm_resource_group.example.location
  resource_group_name       = azurerm_resource_group.example.name
  dns_prefix                = "myakscluster"
  oidc_issuer_enabled       = true
  workload_identity_enabled = true

  default_node_pool {
    name       = "default"
    node_count = 1
  }

  identity {
    type = "SystemAssigned"
  }
}

Step 4: Deploy Argo CD extension for AKS​

Okay, the moment we've all been waiting for, deploying the Argo CD extension for AKS!

Before we can deploy the extension, we need to create the necessary configuration settings for the OIDC integration with Microsoft Entra ID. This includes the OIDC configuration and the RBAC policy mappings.

Add the following code to your main.tf file to create more local variables for the OIDC configuration and RBAC policy mappings:

locals {
  oidc_config = <<EOT
name: Microsoft Entra ID
issuer: https://login.microsoftonline.com/${data.azuread_client_config.current.tenant_id}/v2.0
clientID: ${azuread_application.example.client_id}
azure:
  useWorkloadIdentity: true
requestedIDTokenClaims:
  groups:
    essential: true
requestedScopes:
  - openid
  - profile
  - email
EOT

  policy_csv = <<EOT
g, "${data.azuread_group.example.object_id}", role:admin
EOT
}

This OIDC configuration specifies the issuer URL for Microsoft Entra ID, the client ID of our application, and the requested scopes and claims. The RBAC policy mapping specifies that members of the Microsoft Entra ID group we referenced earlier will be assigned the "admin" role in Argo CD.

With the configuration settings in place, we can now deploy the Argo CD extension for AKS. The nice thing here is that even though the Argo CD extension is in public preview, we can deploy it using the azurerm_kubernetes_cluster_extension resource in Terraform since the underlying call to Azure Resource Manager does not require any special handling for preview extensions.

Add the following code to your main.tf file to deploy the Argo CD extension for AKS:

resource "azurerm_kubernetes_cluster_extension" "argocd" {
  name           = "argocd"
  cluster_id     = azurerm_kubernetes_cluster.example.id
  extension_type = "Microsoft.ArgoCD"
  release_train  = "Preview"

  configuration_settings = {
    "azure.workloadIdentity.enabled"          = "true"
    "azure.workloadIdentity.clientId"         = azuread_application.example.client_id
    "azure.workloadIdentity.entraSSOClientId" = azuread_application.example.client_id
    "redis-ha.enabled"                        = "false"
    "global.domain"                           = "localhost:9000"
    "configs.cm.admin\\.enabled"              = "false"
    "configs.cm.oidc\\.config"                = local.oidc_config
    "configs.rbac.policy\\.csv"               = local.policy_csv
  }
}

You might be asking, what about the version of the Argo CD extension? Since this extension is managed by Azure, we can omit the version and let Azure deploy the latest version and automatically upgrade it when new versions are available. This is one of the benefits of using an Azure-managed extension as it reduces the operational overhead of managing the extension and ensures that you are always running a secure and up-to-date version.

Step 5: Run Terraform and verify​

We're nearly there! The last step is to add some output values to our main.tf file so that we can easily retrieve the resource group name and AKS cluster name after the deployment. Then we'll run terraform init and terraform apply to create all the resources. Once the deployment is complete, we can retrieve the credentials for our AKS cluster and verify that we can sign in to Argo CD using Microsoft Entra ID.

Add the following code to your main.tf file to create the output values:

output "rg_name" {
  description = "The name of the Resource Group"
  value       = azurerm_resource_group.example.name
}

output "aks_name" {
  description = "The name of the AKS cluster"
  value       = azurerm_kubernetes_cluster.example.name
}

Open a terminal, navigate to the directory containing your main.tf file, and run the following commands:

terraform init
terraform apply

The terraform init command will initialize the Terraform working directory and download the necessary providers, while the terraform apply command will create all the resources defined in your main.tf file. Review the plan output and type yes to proceed with the deployment.

Once the deployment is complete, run the following command to retrieve the credentials for your AKS cluster:

az aks get-credentials --resource-group $(terraform output -raw rg_name) --name $(terraform output -raw aks_name)

Now that we have the credentials for our AKS cluster, we can verify that we can sign in to Argo CD using Microsoft Entra ID. Since we set the global.domain to localhost:9000, we will need to port-forward the Argo CD server to access it locally. Run the following command to port-forward the Argo CD server:

kubectl port-forward svc/argocd-server -n argocd 9000:443

Open a web browser and navigate to https://localhost:9000. You should be presented with a warning about an untrusted certificate. This is expected since we're using a self-signed certificate for local testing. Proceed past the warning to see the Argo CD sign-in page. Select the Log in via Microsoft Entra ID button and single sign-on (SSO) should kick in.

Navigate to the "User Info" page and you should see your Microsoft Entra ID user information along with the group claims.

If you have the Argo CD CLI installed, you can also verify the sign-in and view user information from the command line as well.

Keep the port-forwarding session running and open a new terminal window, then run the following command to sign in to Argo CD using SSO:

argocd login localhost:9000 --sso --insecure

You should see a browser window open to complete the authentication flow. After you sign in, you can run the following command to view your user information:

argocd account get-user-info

You will see the same user information and group claims as you saw in the web UI, confirming that the OIDC integration with Microsoft Entra ID is working correctly.

Congratulations! You have successfully secured Argo CD with Microsoft Entra ID for authentication and authorization. You can now manage your Kubernetes applications using Argo CD with the confidence that your deployment is secure and integrated with Azure services.

Cleanup​

When you're done experimenting, you can tear down all the resources created in this walkthrough by running:

terraform destroy

Review the plan output and type yes to confirm. This will remove the AKS cluster, Argo CD extension, Microsoft Entra ID application, and all related resources.

Summary​

In this post, we walked through how to secure Argo CD with Microsoft Entra ID for authentication and authorization. We covered the steps to create a Microsoft Entra ID application for Argo CD, configure OIDC settings, create RBAC policy mappings, deploy the Argo CD extension for AKS, and verify the integration. By leveraging Microsoft Entra ID, you can enhance the security of your Argo CD deployment and take advantage of Azure's identity and access management features.

What's next?​

• Expose Argo CD with an Ingress controller and a real TLS certificate
• Deploy a sample GitOps application through Argo CD to see it in action
• Add granular RBAC roles for different teams in your organization
• Explore the Argo CD extension docs for more tutorials

Tried this out? We'd love to hear how it went — drop us feedback in GitHub Discussions or open an issue if you run into anything.

Happy deploying!

Resources​

• Full Terraform code for this walkthrough
• Announcing public preview of Argo CD extension on AKS and Azure Arc-enabled Kubernetes - Microsoft Tech Community
• Argo CD documentation
• Argo CD Microsoft Entra ID integration
• Argo CD RBAC documentation
• Workload identity for AKS
• Terraform on Azure

• Enable broad, low-friction access to AI services
• Prevent a single application from exhausting shared quotas

This article describes a platform-oriented approach to controlling AI spend using Azure Kubernetes Application Network (AppNet) and agentgateway. By leveraging workload identity already present in the network, you can enforce per-application, token-based rate limiting without issuing API keys to every application.

By adopting this platform-oriented approach, you gain centralized control over AI spending, eliminate secrets distribution, and improve operational efficiency.

Azure Kubernetes Application Network (currently in Public Preview) is Azure's fully managed L7 network for AKS, providing security, observability, and control for your L7 network out of the box. This article focuses on AppNet's secure, automatic mTLS authentication.

The challenge with shared AI quotas​

A typical AI inference workflow looks like this:

1. Create an AI or model provider account.
2. Distribute a shared API key.
3. Allow applications to call the service.

This approach works initially, but as usage grows, teams often encounter a shared failure mode:

403: Insufficient Quota

When the quota is exhausted:

• All applications are affected simultaneously.
• It is difficult to identify which workload consumed the quota.
• One misbehaving application can impact the entire platform.

Why per-application API keys don’t scale​

A common mitigation is to issue one API key per application, each with an independent quota. While this improves attribution, it introduces new operational challenges:

• Manual key provisioning by platform engineers
• Secret distribution and rotation
• Slower onboarding for new applications
• Increased operational overhead

Instead of removing blockers, the platform becomes one.

A platform-oriented solution​

A more scalable approach is to shift rate limiting out of application code and into the platform layer.

The key idea is: Identify applications by identity, not by secrets.

Azure Kubernetes Application Network automatically establishes workload identity through mutual TLS (mTLS) using Istio's ztunnel proxy. By enforcing policy based on this identity, the platform can apply per-application limits transparently.

This solution combines AppNet's automatic mTLS identity on all traffic with agentgateway's ability to define Token Rate Limiting buckets with CEL expressions to accomplish per-application Token budgets. By configuring agentgateway to terminate mTLS directly, we're able to have it participate in the network just like an Istio waypoint would, which gives it direct access to the TLS identity on the wire.

In contrast with our initial scenario, the Azure Foundry API Key is only accessible to the agentgateway, so application teams don't touch any secrets, while AppNet provides per-application identity information on the wire.

Deep Dive: Configure Application Network and agentgateway for token-based rate limiting​

Let's look at the key components that make up our rate limit. For step-by-step setup details, see the detailed configuration gist. First, let's configure agentgateway to interoperate with AppNet, which exposes an Istio-compliant control plane:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: agentgateway
spec:
  deployment:
    spec:
      template:
        metadata:
          labels:
            istio.io/dataplane-mode: none
            networking.istio.io/tunnel: http
  istio: {}
  rawConfig:
    binds:
    - listeners:
      port: 15008
      tunnelProtocol: hboneGateway
  service:
    spec:
      ports:
      - name: hbone
        port: 15008

Next, let's create a policy to perform token-based rate limiting on this gateway's traffic, using the AppNet mTLS identity as the bucket key.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: minute-token-budget
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: gateway
  traffic:
    rateLimit:
      global:
        backendRef:
          group: ""
          kind: Service
          name: ratelimit
          namespace: ratelimit
          port: 8081
        descriptors:
        - entries:
          - expression: source.identity.namespace
            name: client_ns
          unit: Tokens
        domain: token-budgets

Finally, let's configure our Rate Limiting Server to deny traffic after 100 tokens per application per minute (in reality, we'd need a much bigger budget, but this low budget lets us easily demo exceeding the rate limiter).

apiVersion: v1
data:
  config.yaml: |
    domain: token-budgets
    descriptors:
      # Rate limit by client namespace
      - key: client_ns
        rate_limit:
          unit: minute
          requests_per_unit: 100
kind: ConfigMap
metadata:
  name: ratelimit-config

Now that we've configured our rate limiter, let's send some completion requests to Azure Foundry to see it in action. Full test instructions are available in the details gist:

curl gateway.default/v1/chat/completions -i -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }'
HTTP/1.1 200 OK
...(succeeds several times)
curl gateway.default/v1/chat/completions -i -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }'
HTTP/1.1 429 Too Many Requests
content-length: 0
date: Fri, 03 Apr 2026 22:59:18 GMT

Once we've exhausted our token budget, all requests from httpbin to Azure Foundry will be blocked by agentgateway until our budget resets in 1 minute. Requests from other applications can proceed without being blocked, because each application has its own rate limit bucket, keyed on AppNet identity.

Conclusion​

By adopting this platform-oriented approach, you gain centralized control over AI spending, eliminate secrets distribution, and improve operational efficiency. Applications gain transparent rate limiting without code changes, while platform teams reduce overhead and enforce fair resource allocation across the organization. This is just one of the many ways you can benefit from Application Network, built on Istio's Ambient Mode, with readily available open source tools like agentgateway. To learn more, see Application Network documentation and agentgateway documentation.

Introduction​

Earlier parts of this series used runtimes such as Ollama and vLLM to quickly enable local LLM inference and iterate on model behavior. In this post, you move to a more infrastructure‑centric deployment model using NVIDIA Triton Inference Server with the TensorRT‑LLM backend and an instruction‑tuned model. This approach prioritizes predictable performance, efficient GPU utilization, and long‑running inference services, making it well suited for edge and on‑prem environments where capacity is fixed and inference must operate reliably as part of the platform.

TensorRT‑LLM deployment pipeline on Triton​

Unlike Ollama, vLLM, or ONNX-based models, where inference engines initialize at runtime, TensorRT-LLM follows an explicit build-then-serve pipeline. The model is compiled into a GPU-specific engine ahead of time, and Triton uses this engine for runtime inference. Once a TensorRT‑LLM engine is built, it can be deployed to any node as long as the hardware and software environment match the conditions it was built for. At a high level, the pipeline consists of three phases:

For reference​

• TensorRT‑LLM overview
• TensorRT‑LLM engine build workflow
• Triton model repository and ensemble models
• TensorRT‑LLM Triton backend
• Triton Inference Server architecture
• Performance and batching concepts

Why this pipeline matters​

This compile-then-serve model differentiates TensorRT-LLM from earlier approaches in this series. By shifting optimization and hardware specialization to build time, TensorRT-LLM delivers deterministic performance and efficient GPU utilization, which are key requirements for edge and on-prem AI deployments.

TensorRT-LLM build and deployment directory structure​

The following directory structure represents how artifacts flow through the build and deployment phases when using TensorRT-LLM with Triton. This structure helps visualize how model assets, hardware-specific artifacts, and runtime configuration are separated.

/models
├── qwen-raw
│   ├── model.safetensors
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   └── config.json
│
├── model_repository
│   ├── tllm_checkpoint_qwen_7b_int41
│   │   ├── config.json
│   │   ├── *.ckpt
│   │   └── rank0.safetensors
│   └── tllm_engine_qwen_7b_int41
│       ├── config.json
│       └── rank0.engine
│
├── engines
│   └── qwen_7b_int4
│       ├── config.json
│       └── rank0.engine
│
├── tokenizer
│   └── qwen_7b_int4
│       ├── tokenizer.json
│       └── tokenizer_config.json
│
└── triton_model_repo
    ├── ensemble
    │   ├── 1
    │   └── config.pbtxt
    ├── preprocessing
    │   ├── 1
    │   └── config.pbtxt
    ├── tensorrt_llm
    │   ├── 1
    │   └── config.pbtxt
    ├── postprocessing
    │   ├── 1
    │   └── config.pbtxt
    └── tensorrt_llm_bls
        ├── 1
        └── config.pbtxt

Phase 1: Preparation​

Preparing storage for the model repository​

In addition to the prerequisites, you'll need persistent storage for model files. First, create a triton-inference namespace and a PersistentVolumeClaim (PVC) for the model repository.

Save the following YAML as triton-pvc.yaml:

# THE NAMESPACE
# Creates an isolated logical boundary for your Triton resources.
# All subsequent resources must reference this namespace to communicate.
apiVersion: v1
kind: Namespace
metadata:
  name: triton-inference
---
# THE STORAGE (PVC)
# Requests a persistent disk from the cluster to store your model weights.
# This ensures that if the Pod restarts, your downloaded models remain intact.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: triton-model-repository-pvc
  namespace: triton-inference # Ensures the storage is available within your namespace
spec:
  # ReadWriteOnce (RWO) allows the volume to be mounted as read-write by a single node.
  accessModes:
    - ReadWriteOnce

  # Omit storageClassName to use the cluster's default StorageClass.
  resources:
    requests:
      # Allocates 100GB of space. Ensure your underlying disk provider
      # supports this size.
      storage: 100Gi

Apply the manifest and verify the PVC status is Bound (storage provisioned):

kubectl apply -f triton-pvc.yaml
kubectl get pvc -n triton-inference

Deploy a helper pod to download the model​

Next, you'll deploy a temporary helper pod that acts as a provisioner. You'll use this pod to download the model into persistent storage, convert the raw weights into TensorRT-LLM checkpoints, and build a GPU-specific TensorRT engine. This pod requires a GPU during the preparation stage. Save the following YAML as triton-provisioner.yaml.

# THE PROVISIONER POD
# This pod serves as your "workspace" to prepare the environment.
# Its purpose is to download raw models, convert them into checkpoints,
# and compile the optimized TensorRT engines for deployment.
apiVersion: v1
kind: Pod
metadata:
  name: triton-provisioner
  namespace: triton-inference
spec:
  containers:
    - name: provisioner
      # TRT-LLM Image: Contains the necessary NVIDIA libraries (CUDA, TensorRT-LLM)
      # and the Triton server binary required for engine building and testing.
      image: nvcr.io/nvidia/tritonserver:25.01-trtllm-python-py3

      # Keep the pod alive indefinitely so you can 'kubectl exec' into it
      # and run your download/build commands manually or via scripts.
      command: ["/bin/sh", "-c", "sleep infinity"]

      resources:
        limits:
          # A GPU is required for the 'trtllm-build' step to compile
          # the engine for the specific GPU architecture in your cluster.
          nvidia.com/gpu: 1

      volumeMounts:
        # Mount the PVC to /models inside the container.
        # All downloads and engine files saved here will persist across pod restarts.
        - name: model-storage
          mountPath: /models
  volumes:
    - name: model-storage
      # Connects the internal /models path to the 100Gi PVC defined earlier.
      persistentVolumeClaim:
        claimName: triton-model-repository-pvc

Apply the manifest and wait for the triton-provisioner pod to reach Running (you can stop watching with Ctrl+C once it’s running).

kubectl apply -f triton-provisioner.yaml
kubectl get pods -n triton-inference -w

TensorRT-LLM Model Preparation and Triton Repository Assembly​

Save the following script as a prepare-trtllm.sh file. This script prepares a Hugging Face LLM for inference with Triton by downloading the model, converting and quantizing it into a GPU‑specific TensorRT‑LLM engine, assembling the Triton model repository, and generating all required configuration files for end‑to‑end inference.

#!/usr/bin/env bash
set -euo pipefail

# Print a helpful message if any command fails
trap 'echo "FAILED at line $LINENO: $BASH_COMMAND" >&2' ERR

echo "Starting TensorRT-LLM preparation pipeline..."

# Paths
MODELS_ROOT="/models"
RAW_MODEL_DIR="${MODELS_ROOT}/qwen-raw"
CHECKPOINT_DIR="${MODELS_ROOT}/model_repository/tllm_checkpoint_qwen_7b_int41"
ENGINE_BUILD_DIR="${MODELS_ROOT}/model_repository/tllm_engine_qwen_7b_int41"
ENGINES_DIR="${MODELS_ROOT}/engines/qwen_7b_int4"
TOKENIZER_DIR="${MODELS_ROOT}/tokenizer/qwen_7b_int4"
TRITON_REPO_DIR="${MODELS_ROOT}/triton_model_repo"
FILL_TEMPLATE_SCRIPT="/app/tools/fill_template.py"
MODEL_ID="Qwen/Qwen2.5-7B-Instruct"

# Small helper to print what is running
run() {
  echo ""
  echo "RUN: $*"
  "$@"
}

# Validate environment
if [[ ! -d "${MODELS_ROOT}" ]]; then
  echo "FAILED: ${MODELS_ROOT} does not exist. Ensure the PVC is mounted at /models and run inside the provisioner pod." >&2
  exit 1
fi

if ! command -v python3 >/dev/null 2>&1; then
  echo "FAILED: python3 not found. Run inside the TRT LLM provisioner image." >&2
  exit 1
fi

if ! command -v trtllm-build >/dev/null 2>&1; then
  echo "FAILED: trtllm-build not found. Run inside the TRT LLM provisioner image." >&2
  exit 1
fi

# Enter persistent directory
run cd "${MODELS_ROOT}"

# Create required directories (idempotent)
run mkdir -p "${RAW_MODEL_DIR}"
run mkdir -p "${CHECKPOINT_DIR}"
run mkdir -p "${ENGINE_BUILD_DIR}"

echo ""
echo "Downloading model ${MODEL_ID} into ${RAW_MODEL_DIR} ..."

# Download raw model
# Prefer huggingface-cli when available, otherwise fall back to python module invocation
if command -v huggingface-cli >/dev/null 2>&1; then
  run huggingface-cli download "${MODEL_ID}" --local-dir "${RAW_MODEL_DIR}" --exclude "*.bin" "*.pth"
else
  run python3 -m huggingface_hub.cli download "${MODEL_ID}" --local-dir "${RAW_MODEL_DIR}" --exclude "*.bin" "*.pth"
fi

# Convert to quantized checkpoint
run python3 /app/examples/qwen/convert_checkpoint.py \
  --model_dir "${RAW_MODEL_DIR}" \
  --output_dir "${CHECKPOINT_DIR}" \
  --dtype float16 \
  --use_weight_only \
  --weight_only_precision int4

# Build the compressed engine
run trtllm-build \
  --checkpoint_dir "${CHECKPOINT_DIR}" \
  --output_dir "${ENGINE_BUILD_DIR}" \
  --gemm_plugin float16 \
  --max_batch_size 4 \
  --max_input_len 2048 \
  --max_seq_len 3072

echo ""
echo "Finalizing Triton model repository..."

# Create folder layout + copy engine files
run mkdir -p "${ENGINES_DIR}"
run cp -f "${ENGINE_BUILD_DIR}/"* "${ENGINES_DIR}/"

# Copy Triton model repository templates (overwrite allowed)
run mkdir -p "${TRITON_REPO_DIR}"
run cp -rf /app/all_models/inflight_batcher_llm/* "${TRITON_REPO_DIR}/"

# Verify files copied
run ls -la "${TRITON_REPO_DIR}"

# Copy tokenizer files
run mkdir -p "${TOKENIZER_DIR}"
run cp -f "${RAW_MODEL_DIR}/"tokenizer* "${TOKENIZER_DIR}/"

# Fill in model configs with fill_template.py
if [[ ! -f "${FILL_TEMPLATE_SCRIPT}" ]]; then
  echo "FAILED: fill_template.py not found at ${FILL_TEMPLATE_SCRIPT}" >&2
  exit 1
fi

export ENGINE_DIR="${ENGINES_DIR}"
export TOKENIZER_DIR="${TOKENIZER_DIR}"
export MODEL_FOLDER="${TRITON_REPO_DIR}"
export TRITON_MAX_BATCH_SIZE="1"
export INSTANCE_COUNT="1"
export MAX_QUEUE_DELAY_MS="0"
export MAX_QUEUE_SIZE="0"
export FILL_TEMPLATE_SCRIPT="${FILL_TEMPLATE_SCRIPT}"
export DECOUPLED_MODE="false"
export LOGITS_DATATYPE="TYPE_FP32"

# Basic validation that expected config files exist before patching
for f in \
  "${MODEL_FOLDER}/ensemble/config.pbtxt" \
  "${MODEL_FOLDER}/preprocessing/config.pbtxt" \
  "${MODEL_FOLDER}/tensorrt_llm/config.pbtxt" \
  "${MODEL_FOLDER}/postprocessing/config.pbtxt" \
  "${MODEL_FOLDER}/tensorrt_llm_bls/config.pbtxt"
do
  if [[ ! -f "$f" ]]; then
    echo "FAILED: expected config file missing: $f" >&2
    exit 1
  fi
done

# Update ensemble config
run python3 "${FILL_TEMPLATE_SCRIPT}" -i "${MODEL_FOLDER}/ensemble/config.pbtxt" \
  "triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},logits_datatype:${LOGITS_DATATYPE}"

# Update preprocessing config
run python3 "${FILL_TEMPLATE_SCRIPT}" -i "${MODEL_FOLDER}/preprocessing/config.pbtxt" \
  "tokenizer_dir:${TOKENIZER_DIR},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},preprocessing_instance_count:${INSTANCE_COUNT}"

# Update tensorrt_llm config
run python3 "${FILL_TEMPLATE_SCRIPT}" -i "${MODEL_FOLDER}/tensorrt_llm/config.pbtxt" \
  "triton_backend:tensorrtllm,triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},decoupled_mode:${DECOUPLED_MODE},engine_dir:${ENGINE_DIR},max_queue_delay_microseconds:${MAX_QUEUE_DELAY_MS},batching_strategy:inflight_fused_batching,max_queue_size:${MAX_QUEUE_SIZE},encoder_input_features_data_type:TYPE_FP16,logits_datatype:${LOGITS_DATATYPE}"

# Update postprocessing config
run python3 "${FILL_TEMPLATE_SCRIPT}" -i "${MODEL_FOLDER}/postprocessing/config.pbtxt" \
  "tokenizer_dir:${TOKENIZER_DIR},triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},postprocessing_instance_count:${INSTANCE_COUNT}"

# Update BLS config
run python3 "${FILL_TEMPLATE_SCRIPT}" -i "${MODEL_FOLDER}/tensorrt_llm_bls/config.pbtxt" \
  "triton_max_batch_size:${TRITON_MAX_BATCH_SIZE},decoupled_mode:${DECOUPLED_MODE},bls_instance_count:${INSTANCE_COUNT},logits_datatype:${LOGITS_DATATYPE}"

echo ""
echo "Triton model repository templates have been successfully updated."
echo "TensorRT-LLM preparation completed successfully."
echo "Stop the provisioner pod after this to release the GPU for the Triton server."