Our vision for Azure Kubernetes Service is simple: customers should be able to run AKS wherever their applications and data need to be.

Today, we are announcing the public preview of AKS on bare metal Ubuntu. Customers can now deploy AKS directly on their Ubuntu infrastructure, without installing or operating a hypervisor between Kubernetes and the underlying hardware.

This release extends the AKS experience to organizations that need Kubernetes close to their applications, devices, and data. It is designed for environments such as retail locations, factories, branch sites, telecommunications infrastructure, and on-premises AI systems, where performance, latency, hardware access, or data residency are critical.

Earlier this year, we introduced AKS running directly on bare metal Azure Linux through Azure Local small form factor. Support for Ubuntu expands that vision by giving customers another way to run AKS on infrastructure they own while maintaining a consistent Kubernetes and Azure management experience.

Run AKS directly on Ubuntu infrastructure​

Ubuntu is widely deployed across enterprise datacenters, edge environments, and AI infrastructure. Many organizations already have physical Ubuntu servers, established operational practices, and applications built and validated for the Ubuntu ecosystem.

AKS on bare metal Ubuntu allows customers to build on those existing investments. Platform teams can deploy and manage AKS directly on Ubuntu servers, while developers continue using standard Kubernetes APIs and tools.

Removing the virtualization layer also enables Kubernetes workloads to use the available CPU, memory, storage, networking, and accelerator capacity directly. This can improve infrastructure efficiency and provide the hardware access required by performance-sensitive applications.

The Kubernetes control plane, worker capacity, applications, and data remain in the customer’s environment, helping organizations address requirements for low latency, local processing, data sovereignty, and operation in distributed locations.

Built for AI infrastructure choice​

AI applications have diverse infrastructure requirements.

Customers might already own GPUs or specialized accelerators. They might need a particular driver version, device plugin, container runtime, or toolkit for their models and applications. The right configuration can vary significantly based on the hardware, AI framework, and workload.

AKS on bare metal Ubuntu gives customers control over that infrastructure stack. They can select hardware based on workload requirements and configure the supporting GPU software, including:

• GPU drivers
• Kubernetes device plugins
• Container runtimes
• AI frameworks, libraries, and toolkits

This flexibility allows organizations to align Kubernetes with their existing hardware investments and application dependencies rather than adopting a predefined GPU configuration.

Direct access to local hardware and data makes AKS on bare metal Ubuntu well suited for AI inference, computer vision, industrial automation, intelligent retail, telecommunications, and data processing at the edge.

Get started with a single command​

The fastest way to try AKS on bare metal Ubuntu is to run a single command on a customer-owned Ubuntu machine:

curl -sSL https://aka.ms/aksbm | bash -s -- -s <subscription-id> -t <tenant-id>

This end-to-end quickstart script maintained by the AKS team installs the required Azure tooling, signs the machine into Azure, connects it to Azure Arc if it is not already connected, configures the required Azure providers, and deploys an AKS cluster directly onto the physical server.

Azure CLI​

The public preview provides a streamlined Azure CLI path from a provisioned, Arc enabled Ubuntu machine to a running AKS cluster. With a single Azure Resource Manager deployment command, customers can create the required Azure resources and deploy AKS directly on their physical server.

az aksarc deploy --resource-group <resource-group-name> --arc-machine-names <machine-name>

Consistent management through Azure​

Although applications and data run locally, customers can use familiar AKS and Azure CLI workflows to create and manage their clusters.

AKS on bare metal Ubuntu provides a consistent foundation for:

• Cluster creation and lifecycle management
• Kubernetes upgrades
• Identity and access control
• Governance and policy
• Monitoring and observability
• Security management
• Application delivery using GitOps

This gives platform teams centralized visibility and governance without requiring applications or data to leave the customer’s environment. Developers continue using familiar Kubernetes APIs and tools, while infrastructure teams gain an AKS experience that extends to their Ubuntu infrastructure.

Available now in public preview​

AKS on bare metal Ubuntu is now available in public preview. Customers can deploy AKS directly on supported, customer-owned Ubuntu servers without a hypervisor and manage their clusters using familiar AKS, Azure, and Kubernetes tools.

This preview expands where organizations can run AKS while retaining control over their hardware, GPU software stack, applications, and data.

Get started at aka.ms/aks-bare-metal.

It is easy to show a healthy Kubernetes cluster. The harder and more important moment is when a developer opens an application and asks, “Why is my pod broken, and how do I fix it?”

That question shaped AKS desktop v0.10.0. This is our biggest release so far, not because it has the longest list of features, but because it represents where we believe AKS desktop can provide the most value. We are doubling down on AI-assisted troubleshooting and making it easier to move from seeing a problem to understanding what happened and deciding what to do next.

Start with the problem, not the tool​

Troubleshooting Kubernetes can require a lot of context. A pod might be restarting, but the cause could be its configuration, a missing dependency, a failed health check, insufficient resources, or something elsewhere in the application.

Finding the answer often means moving between resource views, logs, events, metrics, YAML, and documentation. You need to know where to look before you can begin making progress.

We want AKS desktop to shorten that path by bringing the investigation together without requiring you to switch between tools.

With the AI Assistant, you can start from the resource that is showing a problem and ask a direct question. The assistant can use the Kubernetes context available in AKS desktop—including resource state, logs, events, and metrics—to help explain what is happening and recommend the next troubleshooting steps.

The goal is not to replace the developer or operator making the decision. It is to help them get oriented faster. Instead of beginning with “Which command should I run?” they can begin with “Why is this failing?”

For example, if a pod is repeatedly restarting, the assistant can help bring together the warning events, recent logs, and workload configuration that might explain the failure. From there, the user can review the evidence, understand the recommendation, and decide how to proceed.

Watch the walkthrough: See how AKS desktop helps investigate and fix a broken Kubernetes app in Debug Kubernetes with AI: Fix a Broken App in AKS desktop.

Make the first investigation easier​

We also heard that an AI troubleshooting experience is only useful if it is easy to start using.

AKS desktop can now automatically detect GitHub Copilot as an AI provider. To use Auto Detect, install the GitHub CLI, sign in to a GitHub account with an active GitHub Copilot subscription, and let AKS desktop detect the authenticated session and available model catalog—including supported Claude models.

You can also connect directly to Anthropic and use Claude with your own API key. AKS desktop continues to support other providers, including Azure OpenAI, OpenAI, Google Gemini, Mistral AI, DeepSeek, and locally hosted or OpenAI-compatible models.

This flexibility lets individuals and organizations use the provider that fits their existing subscriptions, model preferences, compliance requirements, and data residency needs.

Whichever provider you choose, the important part is the connection between the model and the environment you are already investigating. The assistant can work with the current cluster context instead of relying only on a manually written description of the problem.

It also stays within the permissions of the active Kubernetes identity. Users choose whether to enable Kubernetes API requests, and the assistant does not receive broader cluster access simply because AI is being used.

Bring more of your operational knowledge into the experience​

Troubleshooting is rarely one-size-fits-all. Teams develop their own checks, conventions, tools, and ways of diagnosing common failures.

Skills are reusable packages of instructions and domain knowledge that help an AI assistant handle specialized tasks. AKS desktop enables the Microsoft-authored AKS skill by default, giving the AI Assistant AKS-specific guidance for troubleshooting, networking, security, upgrades, and day-to-day operations. Teams can also enable additional skill repositories or add their own to reflect the practices and knowledge specific to their environments.

We are also introducing support for Model Context Protocol (MCP) servers, which can connect the assistant to additional tools and data sources. These integrations are opt-in, and users can control approval for actions requested by the assistant.

Proactive Diagnosis takes another step toward reducing the time between a problem appearing and someone beginning an investigation. It can surface recent warning and error events with recommendations, giving users a place to start before they have manually assembled all the context themselves.

These capabilities are part of the same direction: make troubleshooting more contextual, more repeatable, and easier to begin.

Keep AKS desktop focused while making it extensible​

AI troubleshooting is the center of this release, but teams can also extend AKS desktop with workflows that fit their environments.

AKS desktop is built on the open-source Headlamp project, and this release includes an integrated Plugin Catalog (preview). You can browse, install, update, and manage plugins directly in AKS desktop instead of setting up each integration separately.

The initial catalog includes plugins for Flux GitOps workflows, cert-manager certificate management, and Kubernetes Event-driven Autoscaling (KEDA). Headlamp plugins can also add dashboards, resource views, visualizations, actions, and integrations with external operational tools.

The catalog initially highlights AKS desktop optimized plugins that have passed Microsoft accessibility and localization checks. Turn off that filter to explore Official plugins published by the upstream Headlamp project. Catalog availability can change as plugins are added, updated, deprecated, or removed.

Why this release matters (to me 🤗)​

What makes this release important is not any single setting or integration. It is that AKS desktop is becoming more useful in the moment when users need the most help.

When an application is healthy, almost any dashboard can show green status. The real test is what happens when a pod fails, an event raises a warning, or a developer does not know which part of the system to investigate first.

This is our biggest release so far because it moves us closer to a simpler troubleshooting workflow: find the problem, bring the relevant context together, understand the likely cause, and take the next step with confidence.

That is the experience we are working to improve, and I hope that matters to you too!

Download the latest version from the AKS desktop Releases page, try the new AI troubleshooting experience, and tell us where it helps—and where we should keep improving—by opening an issue.

Customers have told us that compute location is rarely a simple choice. Sometimes data cannot contractually leave a specific facility. Sometimes the preferred Azure region is constrained even though suitable capacity is available in another Azure region or on-premises. And sometimes a workload needs to run close to the data, users, or devices it serves to avoid extra latency. Today, each of these situations often means waiting, overprovisioning, or operating another Kubernetes cluster: with another control plane, upgrade cycle, and set of policies. Flex nodes for AKS (preview) is designed to bring that distributed compute into one familiar operating model. Below, we'll explore these common scenarios in more detail.

Addressing regional CPU and GPU capacity limits​

GPU-accelerated training and fine-tuning jobs are often blocked not by cost or quota, but by regional SKU availability - a familiar SkuNotAvailable or AllocationFailed error when a specific VM size is exhausted in a region. Historically, resolving this meant over-provisioning in advance or standing up and operating a second cluster in a region where capacity exists.

With flex nodes, cluster operators can register capacity from another Azure region, an edge location, or an on-premises data center as a node pool that joins their existing AKS cluster. The scheduler treats these nodes like any other, respecting the labels, taints, and tolerations already defined for the cluster, so pending workloads can be placed on available capacity without standing up and maintaining a parallel cluster and its associated CI/CD or identity configuration.

Aggregating compute across regions under a single control plane​

Many organizations run compute across Azure regions and private data centers but manage each location as a separate cluster stitched together with GitOps and service mesh tooling. Flex nodes bring that distributed capacity into a single AKS control plane, supporting a range of Azure authentication methods like managed identity, Azure Arc, and service principal. Operators retain a single view of the cluster's full footprint through standard tools like kubectl, while workload placement continues to use native Kubernetes primitives, including node affinity, taints and tolerations, and topology spread constraints.

Workload placement for data residency on-premises​

Regulated workloads in healthcare, financial services, and the public sector frequently require that data remain within a specific facility, without managing the lifecycle of a separate on-premises Kubernetes stack. Flex nodes allows on-premises servers to join an AKS cluster as worker nodes. Operators can then use node selectors and taints to pin workloads that must remain local to that hardware, while the remainder of the cluster's workloads run in Azure. Cluster-wide setup remains unified across both environments, and only the data and the compute that processes it stay on-premises.

Get started today​

Flex nodes for AKS is available today in public preview. Documentation, including setup guidance and supported scenarios, is available on Microsoft Learn: flex nodes for AKS overview. As with all AKS preview features, this release is shaped directly by customer feedback. We're particularly interested in hearing about your capacity and networking use cases, additional provider and hardware scenarios, and any gaps you encounter while testing. Please share feedback through the AKS Flex Node GitHub repository and we look forward to seeing what you build!

The combination of ARC and AKS Automatic gives you the power of ARC on a production-ready AKS cluster with managed node pools, built-in monitoring, scaling, security settings, and other defaults that follow AKS best practices. For runner workloads specifically, the pod readiness SLA is also a strong fit because CI/CD jobs depend on predictable pod startup.

ARC runner scale sets work extremely well on AKS Automatic, and the secure-by-default posture of AKS Automatic helps ensure your configuration is optimized and secure. You get the core ARC benefits: GitHub-native CI jobs, Kubernetes-native ephemeral runners, and runners that can live inside your Azure network, including private virtual networks. That means build jobs can reach private endpoints, internal services, and locked-down dependencies without exposing those resources to the public internet.

AKS Automatic is secure by default and has production-minded safeguards enabled. That's a good thing, but it also means the default public ARC Helm chart values need a little tuning. In particular, we need to be explicit about resource requests and image tags.

Let’s walk through the full setup.

What we'll build​

In this lab we'll deploy:

• An AKS Automatic cluster.
• The ARC runner scale set controller.
• A repository-scoped runner scale set.
• A sample GitHub Actions workflow that targets the runner scale set.

The goal is pretty simple. We'll create a workflow where runs-on maps to the ARC runner scale set. When the job queues, ARC creates an ephemeral runner pod on AKS Automatic, the runner connects to GitHub, the job runs, and then the runner goes away.

Prerequisites​

First, make sure you have the Azure CLI, GitHub CLI, kubectl, kubelogin, and Helm installed.

Sign in to Azure and GitHub:

az login
gh auth login

Install or update the AKS preview extension:

az extension add --name aks-preview --upgrade

Check that the local Kubernetes, Azure auth, Helm, and GitHub CLIs are ready:

kubectl version --client
kubelogin --version
helm version
gh auth status

Read the token interactively so it doesn't end up in your shell history:

read -rsp "GitHub token: " GITHUB_TOKEN
echo

Set environment variables​

Now set up a few variables. Update these for your subscription and GitHub repo:

export LOCATION=eastus
export RG=rg-arc-auto-lab
export CLUSTER=arc-auto-lab

export GITHUB_OWNER="github-owner"
export GITHUB_REPO="github-repo"
export GITHUB_CONFIG_URL="https://github.com/${GITHUB_OWNER}/${GITHUB_REPO}"

export RUNNER_SET_NAME=arc-auto-runners
export ARC_SYSTEMS_NAMESPACE=arc-systems
export ARC_RUNNERS_NAMESPACE=arc-runners

The RUNNER_SET_NAME value is important. This is also the value we'll use in the workflow runs-on.

Create or clone the target GitHub repo​

The validation workflow needs to live in a GitHub repository. If you're starting with a new repo for the lab, create it and clone it locally:

gh repo create "${GITHUB_OWNER}/${GITHUB_REPO}" \
  --private \
  --clone

cd "${GITHUB_REPO}"

export GITHUB_BRANCH=$(git branch --show-current)

If you already have a repo, clone it instead:

gh repo clone "${GITHUB_OWNER}/${GITHUB_REPO}"
cd "${GITHUB_REPO}"

export GITHUB_BRANCH=$(git branch --show-current)

Create the AKS Automatic cluster​

First, create a resource group:

az group create \
  --name "${RG}" \
  --location "${LOCATION}"

Now create the Automatic cluster:

az aks create \
  --resource-group "${RG}" \
  --name "${CLUSTER}" \
  --location "${LOCATION}" \
  --sku automatic \
  --no-ssh-key

AKS Automatic uses Azure RBAC for Kubernetes authorization and disables local accounts. Assign your signed-in identity a scoped AKS RBAC role before using kubectl:

AKS_ID=$(az aks show \
  --resource-group "${RG}" \
  --name "${CLUSTER}" \
  --query id \
  --output tsv)

SIGNED_IN_USER_ID=$(az ad signed-in-user show \
  --query id \
  --output tsv)

az role assignment create \
  --assignee "${SIGNED_IN_USER_ID}" \
  --role "Azure Kubernetes Service RBAC Cluster Admin" \
  --scope "${AKS_ID}"

Once the cluster is ready and the role assignment is in place, get credentials using the exec credential flow:

az aks get-credentials \
  --resource-group "${RG}" \
  --name "${CLUSTER}" \
  --format exec \
  --overwrite-existing

# check connectivity
kubectl get nodes

Install the ARC controller​

First up is the ARC controller. For AKS Automatic, we'll give the controller explicit resource requests and limits. Create a values file:

cat > arc-controller-values.yaml <<'EOF'
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi
EOF

Now install the controller:

helm upgrade --install arc \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --namespace "${ARC_SYSTEMS_NAMESPACE}" \
  --create-namespace \
  --wait \
  --timeout 10m \
  -f arc-controller-values.yaml

kubectl rollout status \
  deployment/arc-gha-rs-controller \
  --namespace "${ARC_SYSTEMS_NAMESPACE}" \
  --timeout 5m

Store the GitHub token in Kubernetes​

Now let's create a namespace for the runner resources and store the GitHub token in a Kubernetes secret:

kubectl create namespace "${ARC_RUNNERS_NAMESPACE}" \
  --dry-run=client \
  --output yaml | kubectl apply -f -

printf '%s' "${GITHUB_TOKEN}" | kubectl create secret generic github-pat \
  --namespace "${ARC_RUNNERS_NAMESPACE}" \
  --from-file=github_token=/dev/stdin

Install the runner scale set​

This is the most important part of the setup. The default runner scale set values are close, but AKS Automatic expects a few things to be explicit because Deployment Safeguards block common Kubernetes anti-patterns before they land in the cluster:

1. Resource requests and limits for the listener pod.
2. Resource requests and limits for the runner pod.
3. An explicit, non-latest runner image tag.

Create arc-runner-set-values.yaml:

cat > arc-runner-set-values.yaml <<EOF
githubConfigUrl: ${GITHUB_CONFIG_URL}
githubConfigSecret: github-pat
minRunners: 0
maxRunners: 3

listenerTemplate:
  spec:
    containers:
      - name: listener
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:2.336.0
        command: ["/home/runner/run.sh"]
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
          limits:
            cpu: "2"
            memory: 4Gi
EOF

This example pins ghcr.io/actions/actions-runner:2.336.0, which was the current GitHub runner release when I validated the walkthrough. Runner versions do age out, so check the GitHub runner releases and update the tag when needed.

Now install the runner scale set:

helm upgrade --install "${RUNNER_SET_NAME}" \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --namespace "${ARC_RUNNERS_NAMESPACE}" \
  --create-namespace \
  --wait \
  --timeout 10m \
  -f arc-runner-set-values.yaml

Check the install:

kubectl get pods --namespace "${ARC_SYSTEMS_NAMESPACE}" --output wide

kubectl get autoscalingrunnersets,ephemeralrunnersets,ephemeralrunners \
  --namespace "${ARC_RUNNERS_NAMESPACE}"

Before any jobs run, the controller and listener pods should be running and the AutoscalingRunnerSet should exist. You should not see runner pods yet. ARC creates those only when jobs queue.

Add a validation workflow​

Now add a simple workflow to the target GitHub repo. Create .github/workflows/arc-automatic-validation.yml:

name: ARC AKS Automatic validation

on:
  workflow_dispatch:

jobs:
  validate:
    runs-on: arc-auto-runners
    steps:
      - name: Print runner context
        run: |
          echo "ARC runner reached workflow execution"
          echo "Runner name: $RUNNER_NAME"
          uname -a
          df -h

      - name: Exercise container tooling
        run: |
          docker --version || true
          echo "Validation complete"

If you changed RUNNER_SET_NAME, update runs-on to match it. Then commit and push the workflow:

git add .github/workflows/arc-automatic-validation.yml
git commit -m "Add AKS Automatic ARC validation workflow"
git push --set-upstream origin "${GITHUB_BRANCH}"

Run the workflow​

If this is a new repository or workflow file, give GitHub a few seconds to index the workflow. Confirm that the workflow appears before dispatching it:

gh workflow list \
  --repo "${GITHUB_OWNER}/${GITHUB_REPO}"

Trigger the workflow:

gh workflow run arc-automatic-validation.yml \
  --repo "${GITHUB_OWNER}/${GITHUB_REPO}" \
  --ref "${GITHUB_BRANCH}"

Watch the run:

RUN_ID=$(gh run list \
  --repo "${GITHUB_OWNER}/${GITHUB_REPO}" \
  --workflow arc-automatic-validation.yml \
  --json databaseId \
  --jq '.[0].databaseId')

gh run watch "${RUN_ID}" \
  --repo "${GITHUB_OWNER}/${GITHUB_REPO}" \
  --interval 10 \
  --exit-status

In another terminal, watch ARC scale the runner set:

kubectl get autoscalingrunnersets,ephemeralrunnersets,ephemeralrunners \
  --namespace "${ARC_RUNNERS_NAMESPACE}" \
  --watch

When everything is working, you should see:

• The GitHub Actions job move from queued to in_progress to completed.
• An ephemeral runner object and pod appear in arc-runners.
• The runner pod use your pinned ghcr.io/actions/actions-runner image.
• The workflow log print ARC runner reached workflow execution.

In the validation run, you'll see the job ran on a pod named arc-auto-runners-<id>-runner-<id>, reported runner version 2.336.0, and completed successfully.

Troubleshooting the AKS Automatic-specific bits​

Here are the issues I hit while validating this setup and what they mean.

Missing resource requests​

If the listener or controller is rejected with an error like this:

container <listener> has no resource requests

or:

container <manager> has no resource requests

AKS Automatic Deployment Safeguards blocks these pods because resource requests are a Kubernetes scheduling best practice. Requests give the scheduler enough information to place the controller, listener, and runner pods reliably as ARC scales jobs up and down.

Make sure the controller chart has resources set and the runner scale set chart has listenerTemplate.spec.containers[].resources set.

The latest runner image is blocked​

AKS Automatic Deployment Safeguards can reject the default runner image because it uses a floating latest tag:

Avoiding the latest tag for container: runner

Pin the runner image to an explicit release:

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:2.336.0

The pinned runner version is too old​

Pinning is required, but the pinned version still needs to be current. If the runner logs show:

Runner version vX.Y.Z is deprecated and cannot receive messages.

update the image tag to a current runner release and upgrade the Helm release:

helm upgrade "${RUNNER_SET_NAME}" \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --namespace "${ARC_RUNNERS_NAMESPACE}" \
  --wait \
  --timeout 10m \
  -f arc-runner-set-values.yaml

The job stays queued​

Start by checking the controller, listener, and runner logs:

kubectl logs \
  --namespace "${ARC_SYSTEMS_NAMESPACE}" \
  deployment/arc-gha-rs-controller \
  --tail 200

kubectl logs \
  --namespace "${ARC_SYSTEMS_NAMESPACE}" \
  -l app.kubernetes.io/component=runner-scale-set-listener \
  --tail 200

kubectl logs \
  --namespace "${ARC_RUNNERS_NAMESPACE}" \
  -l app.kubernetes.io/component=runner \
  --tail 200

Also confirm that the workflow runs-on value exactly matches the runner scale set name.

Cleanup​

Remove the Helm releases:

helm uninstall "${RUNNER_SET_NAME}" --namespace "${ARC_RUNNERS_NAMESPACE}"
helm uninstall arc --namespace "${ARC_SYSTEMS_NAMESPACE}"

kubectl delete namespace "${ARC_RUNNERS_NAMESPACE}" "${ARC_SYSTEMS_NAMESPACE}"

Delete the resource group:

az group delete \
  --name "${RG}" \
  --yes \
  --no-wait

If any offline repository runners remain in GitHub after failed experiments, remove them from the repository's Actions runner settings.

That's it. Once the Automatic-friendly values are in place, ARC can scale an ephemeral runner, pick up a queued GitHub Actions job, and run it successfully on AKS Automatic.

Configurable Scheduler Profiles on Azure Kubernetes Service (AKS) let you configure your own scheduling logic: enable specific plugins, adjust plugin priorities, and tune parameter weights. The result: higher node density, better CPU and GPU utilization, and lower infrastructure costs.

You'll learn how the default Kubernetes scheduler places pods, where the defaults fall short, and how to increase node utilization with Configurable Scheduler Profiles on AKS.

1. How does kube-scheduler work?
2. Use Configurable Scheduler Profiles to increase node utilization and operator control
3. Increase AKS cluster CPU utilization up to 85% with Configurable Scheduler
4. Increase AKS cluster GPU or CPU utilization while balancing memory with Configurable Scheduler
5. FAQ: How do Configurable Scheduler Profiles interact with autoscalers?

How does the default Kubernetes scheduler place pods?​

The Kubernetes scheduler operates in two cycles: a synchronous scheduling cycle and an asynchronous binding cycle. The scheduling cycle has two sub-phases, filtering and scoring, and only manages one pod at a time.

1. Filtering phase removes nodes that do not satisfy the pod's scheduling requirements, based on hard and soft constraints.
2. Scoring phase assigns a score for the remaining nodes, based on scheduler plugins and their configured weights, with the highest-scoring node selected as the scheduling target.

After a node is selected, the scheduler enters the binding cycle, which can process multiple pods concurrently. During this phase, the scheduler attempts to bind the pod to the chosen node. If the bind operation fails, the pod is typically re-queued and retried in a subsequent scheduling cycle rather than immediately falling back to the next-highest-scoring node. When filtering and scoring nodes, the default scheduler considers several hard and soft constraints with predefined weights, including (but not limited to):

1. Resource requirements (CPU, memory)
2. Node affinity/anti-affinity
3. Pod affinity/anti-affinity
4. Taints and tolerations
5. TopologySpreadConstraints

For more on how the Kubernetes scheduler works, see the technical blog post from SIG Scheduling contributor and AKS upstream engineer Heba Elayoty, Deep Dive into the Kubernetes Scheduler Framework.

Limitations of the default Kubernetes scheduler​

The default scheduler is primarily designed for general-purpose workloads. It prioritizes nodes with the most available resources using the LeastAllocated scoring strategy. This strategy spreads pods across nodes, even when they could safely be packed more densely. While this works well for many services, the default scheduling criteria, and their fixed priority order, are not suitable for workloads that demand optimizing GPU and CPU utilization. In these scenarios, spreading pods across nodes can lead to fragmented resources, underutilized GPUs, and increased infrastructure cost.

Today, the default scheduler on AKS lacks the flexibility for users to change which criteria should be prioritized, or ignored, in the scheduling cycle because the default scheduler is not accessible to users. This rigidity often forces users to either accept suboptimal placement or manage a separate custom scheduler, both of which increase operational complexity. Starting with Kubernetes v1.33, AKS introduces Configurable Scheduler Profiles - an AKS-managed CRD - that exposes the upstream scheduling framework without maintaining a separate scheduler. Now, users can adjust the NodeResourcesFit plugin from the default configuration to favor nodes with higher utilization to achieve more efficient bin‑packing and reduce infrastructure cost.

Configurable Scheduler Profiles on AKS​

Configurable Scheduler Profiles on AKS allow users to change the default scheduler behavior using the extensibility of the scheduling framework, without the operational overhead of adopting a second scheduler or defining a custom scheduler. You express the intent declaratively, while AKS safely applies and manages the resulting scheduler behavior using a Custom Resource Definition (CRD).

1. AKS begins with the default scheduling configuration.
2. You declare your desired scheduling behavior using a Configurable Scheduler Profile applied as a CRD.
3. The controller overlays the user-defined configuration on top of the base scheduler configuration.
4. The controller produces and maintains a reconciled scheduler ConfigMap representing the combined configuration.
5. The kube-scheduler deployment consumes the controller-managed ConfigMap and applies the updated scheduling behavior.
6. If the scheduler becomes unhealthy, the controller automatically rolls back to the last known good configuration.

A profile is a set of one or more in-tree scheduling plugins and configurations that dictate how to schedule a pod. AKS supports 18 in-tree Kubernetes scheduling plugins.

Increase node utilization and operator control​

In this simple scale-out scenario, manually increasing identical CPU-bound replicas from 8 to 30, the default scheduler distributes pods evenly across nodes.

Configurable Scheduler Profiles configured for bin-packing show a visible consolidation pattern that differs from the default scheduler, and improves capacity for new pods. This shift occurs without changing the workload, node size, or autoscaling behavior - only the scheduler’s scoring logic.

While this experiment uses intentionally simple, CPU-bound containers to isolate scheduling behavior, the placement patterns observed here can be applied to GPU-bound workloads where capacity constraints elevate the need for utilization improvements. In this constrained GPU scenario the bin‑packing scheduler plays a critical role in maximizing usable capacity of 4 GPUs spread across 2 nodes. By consolidating single‑GPU workloads onto the same node, the scheduler avoids idle GPUs and enables all four GPUs to be actively used before new nodes are required.

This change in distribution shape enables downstream efficiencies: improved control for platform engineers, efficient resource usage, and cost optimization that are difficult to achieve when pods are evenly spread. Configurable Scheduler has many use cases and each profile expresses a distinct scheduling intent. The next two sections detail how the scoring strategies, MostAllocated and RequestedToCapacityRatio, are configured to achieve increased utilization outcomes.

Increase AKS CPU utilization​

Using RequestedToCapacityRatio, this bin packing profile is configured to favor nodes within a utilization band of 50—85%, avoiding empty nodes, and severely deprioritizing nearly full nodes at 90% utilization or more, to limit oversaturated nodes. Configure node bin-packing using the RequestedToCapacityRatio strategy to improve utilization and reduce infrastructure costs.

1. RequestedToCapacityRatio scores nodes based on the ratio of requested resources to total node capacity after the pod is hypothetically placed. This strategy enables more fine-grained bin‑packing by allowing operators to define an ideal utilization curve for their nodes rather than simply preferring the most or least utilized nodes.
2. PodTopologySpread is disabled in this profile because bin-packing and zone-spreading are opposing goals and the scheduling logic may prioritize pod spreading. If you need both high utilization and zone resilience, define a new profile to achieve both goals.

By shaping the scoring curve to target a range of 50—85% CPU utilization, users can increase pod density on provisioned nodes while preserving headroom for bursts, background processes, and system components in CPU-bound workloads.

Given this level of configuration detail, RequestedToCapacityRatio is the recommended scoring strategy for node bin‑packing on AKS for production clusters.

apiVersion: aks.azure.com/v1alpha1
kind: SchedulerConfiguration
metadata:
  name: upstream
spec:
  rawConfig: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: cpu-binpacking-scheduler
        plugins:
          multiPoint:
            enabled:
              - name: NodeResourcesFit
            disabled:
              - name: PodTopologySpread
        pluginConfig:
          - name: NodeResourcesFit
            args:
              apiVersion: kubescheduler.config.k8s.io/v1
              kind: NodeResourcesFitArgs
              scoringStrategy:
                type: RequestedToCapacityRatio
                resources:
                  - name: cpu
                    weight: 8
                  - name: memory
                    weight: 1
                requestedToCapacityRatio:
                  shape:
                    - utilization: 0
                      score: 0
                    - utilization: 30
                      score: 8
                    - utilization: 50
                      score: 10
                    - utilization: 85
                      score: 10
                    - utilization: 90
                      score: 3
                    - utilization: 100
                      score: 0

Increase AKS GPU utilization​

When MostAllocated and NodeResourcesBalancedAllocation are combined, the scheduler favors GPU‑bound nodes with balanced CPU and memory usage over nodes with large amounts of unused memory or fragmented resources. This approach reduces fragmented GPU capacity and results in fewer underutilized secondary resources. Configure node bin-packing using the MostAllocated strategy to improve utilization and reduce infrastructure costs.

1. MostAllocated scores nodes based on its current resource utilization, favoring nodes that are already heavily used for the specified resources.
2. RequestedToCapacityRatio lets you define a scoring curve so you can explicitly control preferred utilization ranges and scores nodes based on resource requests relative to the remaining node capacity. This makes MostAllocated more aggressive for consolidation but gives you less explicit control over headroom.
3. PodTopologySpread is disabled in this profile because bin-packing and zone-spreading are opposing goals. Enabling both can result in spreading over consolidation, weakening the intended packing behavior.

NodeResourcesBalancedAllocation complements MostAllocated because it prefers nodes whose CPU and memory utilization stay proportionally balanced, helping reduce bottlenecks caused by asymmetric resource pressure.

This scheduler configuration maximizes GPU utilization by consolidating smaller jobs onto fewer nodes, reducing idle accelerator capacity while maintaining reasonable CPU and memory balance.

apiVersion: aks.azure.com/v1alpha1
kind: SchedulerConfiguration
metadata:
  name: upstream
spec:
  rawConfig: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: gpu-node-binpacking-scheduler
        plugins:
          multiPoint:
            enabled:
              - name: NodeResourcesFit
              - name: NodeResourcesBalancedAllocation
            disabled:
              - name: PodTopologySpread
        pluginConfig:
          - name: NodeResourcesFit
            args:
              apiVersion: kubescheduler.config.k8s.io/v1
              kind: NodeResourcesFitArgs
              scoringStrategy:
                type: MostAllocated
                resources:
                  - name: nvidia.com/gpu
                    weight: 8
                  - name: cpu
                    weight: 1
          - name: NodeResourcesBalancedAllocation
            args:
              apiVersion: kubescheduler.config.k8s.io/v1
              kind: NodeResourcesBalancedAllocationArgs
              resources:
                - name: cpu
                  weight: 1
                - name: memory
                  weight: 1

FAQ​

1. Which bin-packing strategy does AKS recommend to increase node utilization?

   AKS recommends using the scoring strategy RequestedToCapacityRatio because it provides a more granular scoring approach allowing users to define an ideal utilization curve for their respective nodes. For example, this bin packing strategy allows users to configure a target utilization of 85%.

2. How do Configurable Scheduler Profiles interact with autoscalers such as Node Auto Provisioning (NAP), Cluster Autoscaler (CA), and Vertical Pod Autoscaler (VPA)?

   These components are complementary to each other. Configurable Scheduler Profiles influence how pods are placed on nodes, while autoscalers make scaling decisions based on resource utilization and pending pods. Look out for an upcoming blog that details how scheduling constraints affect Node Auto Provisioning.

   • Node Auto Provisioning (NAP) is triggered when pods are unschedulable. If a suitable node already exists, that pod will be scheduled with the defined Configurable Scheduler Profile. If no suitable node exists, NAP provisions new capacity and schedules the pod.
   • Cluster Autoscaler (CA) manages node scale-up and scale-down. On scale-up, CA is triggered when there aren't any suitable nodes available for the pending pod. Using Configurable Scheduler Profiles ensures nodes are only scaled when provisioned resources are no longer suitable. On scale-down, CA is triggered when nodes fall below utilization thresholds, the default is 50%. As active nodes are packed more efficiently, underutilized nodes become easier candidates for removal.
   • Vertical Pod Autoscaler (VPA) optimizes resource utilization patterns in pods. As pods are recreated with updated CPU and memory requests, they are scheduled using the configured scheduler profile, allowing placement decisions to reflect the new resource requirements.

3. What if a resource, like memory, is omitted in the scoringStrategy?

   If a resource is omitted in the scoringStrategy, then that resource will not be considered in the filter or scoring cycles of the defined Configurable Scheduler Profile. If that resource should be considered, but with a reduced influence on the final score, it can be included with reduced weight.

4. Can multiple Configurable Scheduler Profiles be used for different workloads on the same cluster?

   Yes, multiple scheduling profiles can coexist in a single cluster. This allows different placement strategies (for example, cost‑optimized vs. latency‑sensitive workloads) to run side‑by‑side. Visit the documentation for a multiple scheduler profiles example.

   • Multiple profiles can be defined centrally in a single scheduler configuration.
   • Individual workloads select a profile via schedulerName in the pod spec.

5. How do I monitor whether my scheduler profile is improving utilization?

   Monitor these signals to confirm that the scheduler is behaving correctly. Over time, you should see higher average node utilization, reduced variance between nodes, and fewer lightly utilized nodes.

   • Track node‑level utilization metrics, including CPU and memory utilization per node and distribution of pods across nodes, using Azure Monitor for containers, the AKS node viewer tool, or kubectl top nodes for quick validation.
   • Review autoscaler outcomes, looking for fewer scale‑ups during normal load and more decisive scale‑downs after demand drops.
   • Measure cost metrics, such as reduced idle costs when you use the Cost Analysis add-on.

Next steps: Optimize resources with Configurable Scheduler Profiles on AKS​

Configurable Scheduler Profiles give you direct control over pod placement. With these scheduling plugins, your workloads make full use of available GPU capacity, reduce idle costs, and avoid costly overprovisioning.

• For additional guidance and best practices, see kube-scheduler best practices
• Increase node utilization using Configurable Scheduler Profiles
• To schedule and queue batch workloads, install and configure Kueue on AKS.

We are delighted to announce the open-source release of TauGrid. TauGrid combines the tau CLI, Kueue queueing, KubeRay orchestration, GPU-node health monitoring, and observability in a single Kubernetes-native stack. It supports workflows spanning data preparation, distributed training, fine-tuning, and inference.

Instead of building and maintaining these components and the integration between them separately, TauGrid provides a single Helm install with clear ownership boundaries. Platform teams get a unified way to operate the stack, while researchers get one CLI to submit workloads, observe their progress, recover from failures, and retrieve results without interacting with Kubernetes directly.

The project is MIT-licensed and available at github.com/Azure/taugrid, with container images and Helm charts published to Microsoft Container Registry. Full documentation is available at azure.github.io/taugrid.

What is TauGrid?​

TauGrid is an open-source, self-hosted platform for running AI workloads on Kubernetes. It gives teams the key benefits of a managed AI platform (job submission, scalable distributed computing, shared GPU resources, experiment tracking, and operational controls) while running on infrastructure they manage in their own cloud or data center.

Researchers work from a repository and the Tau CLI. Platform teams provide governed workspaces, queues, compute profiles, storage, identity, and observability. TauGrid connects those two experiences across training, fine-tuning, batch inference, and serving. The result is reduced operational complexity for platform teams, faster onboarding for researchers, better GPU utilization through shared queueing, and consistent experiment reproducibility through evidence records.

How it works​

TauGrid uses a tau.yaml configuration file to describe the workload. Here is a GPU training example that runs a PyTorch workload on a single A100:

schema_version: 1
name: aks-gpu-quickstart
run:
  entrypoint: train.py
  workload_kind: rayjob
compute:
  gpus: 1
  workers: 1
  cpus: 16
  memory: 64Gi
runtime:
  image: mcr.microsoft.com/aks/ai-runtime/ray:py3.12-ray2.56.0-cuda13.0
  pip:
    - torch>=2.4.0

Once submitted with tau run, TauGrid resolves platform policy, renders a KubeRay RayJob, and submits it through Kueue. Status, logs, checkpoints, and experiment evidence are tracked throughout the run. Evidence records capture workload metadata, configuration, logs, metrics, checkpoints, and execution history, helping teams reproduce results, troubleshoot failures, and maintain an audit trail.

The same stack covers the full GPU workload lifecycle. This animation shows a workload moving from data preparation through distributed training, fine-tuning, and into a ready inference endpoint, all with one continuous evidence record:

Here is what TauGrid handles at each stage:

When multiple teams share the same cluster, their workloads enter a shared Kueue ClusterQueue. Kueue admits each job based on quota and priority, and Kubernetes places it on healthy GPUs.

Getting started​

You need a Kubernetes 1.30+ cluster, kubectl, Helm 3+, and Git. Install the Tau CLI:

curl -fsSL https://github.com/Azure/taugrid/releases/latest/download/install.sh | sh

export PATH="$HOME/.local/bin:$PATH"
tau version --short

Install TauGrid onto your cluster:

tau cluster install
tau cluster validate installation

Create a workspace and wait for it to be ready:

tau workspace create "taugrid-default" --apply

kubectl wait \
  --for=jsonpath='{.status.phase}'=Ready \
  workspaces.tau.azure.com/taugrid-default \
  --namespace tau-system \
  --timeout=5m

Submit your first workload:

git clone https://github.com/Azure/taugrid.git
cd taugrid

tau run --config examples/cpu-multi-interest-ray/tau.yaml
tau run status cpu-multi-interest-ray --watch
tau run logs cpu-multi-interest-ray -f

This example runs a CPU workload and does not require GPU nodes. For the full step-by-step guide including cluster preparation, workspace configuration, GPU workloads, and handoff to researchers, see the Getting Started on Kubernetes documentation.

The repository also includes these runnable examples:

For more examples and detailed walkthroughs, see the examples documentation.

Looking ahead​

TauGrid is under active development. The full roadmap is maintained in the repository.

Multi-tenant platform capabilities: Support for multiple workspaces with scoped identity, RBAC, and quotas, along with an interactive portal for submitting and managing workloads.

Distributed training workflows: End-to-end examples for PyTorch DDP, FSDP, DeepSpeed, and Hugging Face LoRA/QLoRA, plus a complete dataset lifecycle covering fetch, staging, validation, tokenization, and registration.

Inference workflows: Production-oriented serving examples for vLLM, SGLang, and TensorRT-LLM.

Multi-cluster and multi-cloud execution: Cross-cloud workload execution with portable data, checkpoints, and artifacts, supported by provider-agnostic observability and cost attribution.

TauGrid will continue to focus on the workflow and workload lifecycle layer. Cluster provisioning, pod scheduling, quota enforcement handled by Kubernetes and Kueue, framework internals, and model code remain outside its scope.

What's next?​

Install the CLI, point it at any Kubernetes 1.30+ cluster, and submit your first workload. We are building TauGrid in the open and want your input: file an issue when something breaks, open a pull request to fix a bug or add a feature, share feedback on the lifecycle contract, or tell us how TauGrid fits (or does not fit) into your existing platform.

Get started with the documentation, explore the code on GitHub, and check out the contributing guide to learn how to get involved.

When you manage Kubernetes, a few disruption questions come up fast:

• How do I control when scale down happens, or when it should not happen?
• How do I make workload disruption predictable?
• Why won’t NAP scale down my nodes, even with lots of underused capacity?
• Why do upgrades get stuck on certain nodes?

This post focuses on NAP disruption best practices, not workload scheduling tools such as topology spread constraints, node affinity, and taints. For scheduling best practices, see the NAP scheduling fundamentals blog post.

If you’re new to these features, start here. If you already use NAP disruption settings, use this post as a checklist for the behaviors AKS users most commonly ask about.

Two layers of disruption control​

When NAP decides a node (virtual machine) could be removed, two layers of controls determine whether it actually happens:

Workload layer: Pod Disruption Budgets (PDBs)​

PDBs are Kubernetes-native guardrails that limit voluntary evictions of pods. PDBs tell Kubernetes:

"During voluntary disruptions, keep at least N replicas available, or limit max unavailable."

Infrastructure layer: node-level disruption settings​

NAP exposes disruption controls at the node level.

NAP is built on Karpenter concepts and exposes disruption controls on the NodePool:

• Consolidation policy: when NAP is allowed to consolidate
• Disruption budgets: how many nodes can be disrupted at once, and when
• Expire-after: node lifetime
• Drift: replace nodes that are out of date with the desired NodePool configuration

A good operational posture is simple: use PDBs to protect applications and use NAP disruption tools to control the cluster's disruption rate.

NAP overview​

Node auto-provisioning (NAP) provisions, scales, and manages nodes. NAP bases its scheduling and disruption logic on settings from three sources:

• Workload deployment file: for disruption, NAP honors the PodDisruptionBudgets you define here.
• NodePool CRD: lists the allowed virtual machine options, such as size, zones, and architecture, and also defines disruption settings.
• AKSNodeClass CRD: defines Azure-specific settings.

How NAP handles disruption​

NAP honors Kubernetes-native concepts such as PodDisruptionBudgets when making disruption decisions. NAP also includes Karpenter-based concepts such as consolidation, drift, and node disruption budgets.

What disruption means in NAP​

In NAP, disruption typically refers to voluntary actions that delete nodes after draining them, such as:

• Consolidation: delete or replace nodes with better virtual machine sizes to improve compute efficiency and reduce cost.
• Drift: replace existing nodes that no longer match the desired configuration, for example, updated settings in your NodePool and AKSNodeClass CRDs.
• Expiration: replace nodes after a configured lifetime.

These actions differ from involuntary disruptions such as:

• Spot eviction events
• Hardware failures
• Host reboots outside your control

PDBs and Karpenter disruption budgets mainly help with voluntary disruptions. These features do not regulate involuntary disruption, for example, spot VM evictions, node termination events, or node stopping events.

How NAP handles spot nodes​

NAP can provision Azure Spot VMs, which are spare capacity in Azure available at a discount, and can be evicted by Azure once this capacity is needed back. Spot VMs are prone to eviction, by design. We recommend you limit Spot VM use to workloads that can handle disruption, and use on-demand compute for stable capacity needs. Disruption controls detailed in this blog will not stop/control spot eviction, or involuntary disruption. You can use some of these tools like consolidation policies with spot to limit when these spot VMs are disrupted due to node utilization.

When NAP receives a signal that a NAP-managed spot VM is scheduled to be evicted, Node Problem Detector flags the VM(s) as unhealthy, and NAP will begin provisioning a replacement VM.

Pod Disruption Budgets (PDBs): controlling voluntary disruption​

Most NAP disruption problems come from PDBs that are either:

• Too strict: a strong guardrail blocks node drains indefinitely.
• Missing: no guardrail limits disruption.

A good default PDB​

Kubernetes documentation describes minAvailable and maxUnavailable as the two key availability settings for PDBs. You can specify only one per PDB.

Here's an example of a PDB that regulates disruption without blocking scale down, upgrades, and consolidation:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web

Why it works well in practice:

• Consolidation, drift, and expiration can still proceed.
• It helps you avoid large brownouts caused by draining too many replicas at once.
• It reduces the chance of NAP thrashing a service by repeatedly moving too many pods.

The common PDB pitfall: zero voluntary evictions​

If you effectively set zero voluntary evictions, for example, maxUnavailable: 0 or minAvailable: 100%, Kubernetes warns that this can block node drains indefinitely for a node running one of those pods.

This common misconfiguration can cause scenarios such as:

• Node and cluster upgrades fail because nodes do not voluntarily scale down.
• Migration fails.
• NAP consolidation never happens.

This can be intentional for extremely sensitive workloads, but it has a cost. If a node has one of these pods, draining that node can become impossible without changing the PDB or taking an outage. Set some tolerance in these PDB settings and use disruption budgets or maintenance windows to control disruption.

Practical guidance:

• Choose maxUnavailable or maxUnavailable based on your replica count and required service capacity.
• For critical workloads that you do not want to disrupt, zero eviction may be intentional. Be deliberate. When you're ready to allow disruption to these workloads, you may need to change the PDBs in the workload deployment file.
• For multi-replica workloads that can tolerate minor disruption, prefer a small maxUnavailable value, such as 1, instead of zero voluntary disruption.
• Be clear on the tradeoff. Zero tolerance can block upgrades, NAP consolidation, and scale down.

The following example allows one replica to be unavailable during voluntary disruption. It is appropriate only when the workload has enough replicas to preserve its availability target. Choose maxUnavailable or minAvailable based on your replica count and required service capacity.

Controlling consolidation: when vs. how fast​

Two operator intents often get conflated:

• When consolidation is allowed and happens.
• How much disruption can happen concurrently.

Use these settings to control consolidation behavior:

• consolidationPolicy: WhenEmptyOrUnderutilized: triggers when NAP identifies existing nodes as underutilized or empty. NAP runs cost simulations of virtual machine size combinations that best match the current configuration. When it finds a better combination, consolidation starts.
• consolidateAfter: 1d: controls the delay before NAP consolidates underutilized nodes, and works with the consolidationPolicy setting.
• expireAfter: 24h: determines how long nodes in this NodePool CRD can exist. Older nodes are deleted regardless of consolidation policies.

Note: Underutilized rate is not a value you can set. NAP determines it through internal cost simulations.

The following example shows these disruption tools in action:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
  template:
    spec:
      nodeClassRef:
        name: default
      expireAfter: Never

Node disruption budgets: how fast​

NAP exposes Karpenter-style disruption budgets on the NodePool. If you don’t set them, a default budget of nodes: 10% applies. Use budgets to regulate how many nodes are consolidated at a time.

The following example sets the node disruption budget to one node at a time.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    budgets:
    - nodes: "1"

The following example sets the node disruption budget to not disrupt more than 10% of nodes at a time.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    budgets:
    - nodes: 10%

This is often the simplest way to prevent NAP from moving too many nodes at once.

Maintenance windows​

A good practice is to allow some consolidation, but only during a specific time window.

NAP node disruption budgets support schedule and duration, so you can create time-based rules with cron syntax. Define these budgets in the spec.disruption.budgets field of the NodePool CRD.

For example, block disruptions during business hours:

budgets:
- nodes: "0"
  schedule: "0 9 * * 1-5"  # 9 AM Monday-Friday
  duration: 8h

Or allow higher disruption on weekends, and block disruption otherwise:

budgets:
- nodes: "50%"
  schedule: "0 0 * * 6"    # Saturday midnight
  duration: 48h
- nodes: "0"
  schedule: "0 9 * * 1-5"  # 9 AM Monday-Friday
  duration: 8h

Why this matters: It aligns cost optimization, including consolidation, drift, and expiration, with the timeline that works for your workload needs.

To learn more about node disruption budgets, see the NAP disruption documentation.

Observability: verify disruption decisions with events and logs​

Before you change policies, confirm what NAP thinks it's doing:

• View events:
  • run command kubectl get events --field-selector source=karpenter-events
• Or use AKS control plane logs in Log Analytics, filtered for karpenter-events

This helps you distinguish between these cases:

• NAP wants to disrupt nodes, but PDBs or disruption budgets block it.
• NAP is not trying to disrupt nodes because the consolidation policy does not allow it.
• NAP can't replace nodes because provisioning is failing.

For more on monitoring NAP logs and metrics, see the NAP observability documentation.

Common disruption pitfalls​

Symptom: NAP won’t consolidate or drains hang forever​

Behavior: Nodes do not scale down for consolidation or updates.

Cause:

• PDBs effectively allow zero voluntary evictions, for example, maxUnavailable: 0 or minAvailable: 100%.
• Too few replicas exist to satisfy the PDB during drain.

Fix:

• Relax PDBs, for example, maxUnavailable: 1, or increase replicas.
• If a workload truly must not be disrupted, accept that nodes running it will not consolidate. This also increases the risk of update failure. A strict 100% available PDB can cause scale down and update failures.

Symptom: NAP disrupts nodes too often or too much at once​

Behavior: NAP consolidates too often or voluntarily disrupts too many nodes at once.

Cause:

• No guardrails exist for node disruption behavior, such as PDBs or node disruption budgets.
• No maintenance window exists for scheduled disruption times.

Fix:

• Add PDBs that regulate disruption pace.
• Add NodePool disruption budgets. Start with nodes: "1" or a small percentage.
• Add time-based budgets so disruption happens when you want it.

Symptom: node disruption happens at the wrong time​

Behavior: Disruption happens during inconvenient times, such as work hours or peak usage windows.

Cause:

• No time-based budgets or maintenance window exists.

Fix:

• Add Karpenter disruption budgets to block disruption during business hours.
• Combine a maintenance window with a small allowed-disruption budget outside the window.

Next steps​

1. Try NAP today: Check out the Enable Node Auto Provisioning steps.
2. Learn more: Visit the AKS operator best-practices guidance.
3. Troubleshoot NAP: For more common issues visit our NAP troubleshooting guide
4. Share feedback: Open issues or ideas in AKS GitHub Issues.
5. Join the community: Subscribe to the AKS Community YouTube and follow @theakscommunity on X.

When you autoscale LLM inference on Kubernetes, cold-start time is dominated by one thing: getting those tens of gigabytes of model weights off storage and into GPU memory. The naive path involves downloading all the model weights to local disk, then loading them into the GPU. However, this serializes two slow copies back-to-back and leaves the network idle while the GPU waits.

The RunAI Model Streamer collapses that into a single streaming step: it reads tensors concurrently from object storage and feeds them into GPU memory as they arrive, saturating the available bandwidth instead of staging everything on disk first. vLLM wires it in behind --load-format runai_streamer. For production inference on AKS, that translates to pods that reach a ready, serving state faster, which means quicker recovery from node failures, snappier rollouts, and autoscaling that can add replicas in time to absorb a traffic spike instead of lagging behind it.

Support for Azure Blob Storage was recently added to the RunAI Model Streamer. As of vLLM v0.18.0 (vllm-project/vllm#34614) and runai-model-streamer v0.15.6 (run-ai/runai-model-streamer#116), the az:// scheme is now supported out of the box. A stock vllm/vllm-openai image can stream from Blob with nothing more than an environment variable and a workload-identity binding.

This post walks the whole thing end to end on AKS:

1. An AKS cluster with OIDC + workload identity, and a fully managed A100 GPU node pool (AKS installs and maintains the NVIDIA driver, device plugin, and DCGM exporter, no need to install the NVIDIA GPU operator).
2. A premium block-blob storage account to hold the weights.
3. Workload identity that lets pods read and write Blob with a Microsoft Entra ID token instead of a storage key.
4. An in-cluster upload Job that pulls microsoft/phi-4 from HuggingFace and pushes it to Blob, running inside Azure, so it uses Azure's backbone bandwidth rather than your home uplink. Such jobs need to be run once per model.
5. A vLLM Deployment that streams those weights straight from az:// at startup.

:::note Why upload from a Job instead of your laptop? Multi-gigabyte model weights uploaded from a workstation are gated by your upstream bandwidth and can take hours. A Job in the same region as the storage account moves the same bytes over Azure's network in minutes, and it authenticates with workload identity so there are no keys to copy around. :::

Why stream instead of download-then-load?​

The default way vLLM loads a model is a two-step relay: download all the model weights from the source (HuggingFace Hub or object storage) onto the node's local disk, then read them back off disk to load them into GPU memory. The RunAI streamer replaces both halves with a single concurrent stream from object storage straight into GPU memory.

To put real numbers on it, we timed both paths for microsoft/phi-4 (about 27.4 GiB, ~29 GB of weights) on the same single, A100 node (Standard_NC24ads_A100_v4) with vLLM v0.23.0, the exact setup this post walks you through:

The single most comparable number is the load step itself, reading the same ~29 GB (about 27.4 GiB) into GPU memory: ~74 s off local disk versus ~10 s streamed from Blob, roughly 7× faster, because the streamer overlaps the read with the GPU copy and reads with high concurrency instead of shard-by-shard. Add the download the default path can't avoid and the end-to-end gap widens to ~187 s vs ~10 s. (The download half in particular is network-dependent and will vary run to run; the load-step comparison is the stable, apples-to-apples one.)

Three things make the default path slow, and the streamer fixes each one:

• Two serial passes instead of one. With download-then-load, nothing reaches the GPU until the last byte has landed on disk. The GPU sits idle through the entire download, and the network sits idle through the entire disk-to-GPU load, the two slow copies never overlap. The streamer pipelines them: a tensor is loading onto the GPU while the next ones are still in flight from storage.
• A wasted round-trip through disk. Staging to disk writes tens of gigabytes and immediately reads them back, a second full pass over the data that exists only to bridge the two steps. Even on fast local NVMe that's pure overhead; on network-attached or smaller disks it's worse, and large model weights can fill the disk outright. Streaming never touches disk.
• No concurrency. A plain download and a plain disk read each tend to move data in a single stream, leaving bandwidth unused. The streamer issues many parallel reads against object storage and feeds the GPU concurrently, keeping the link saturated, which is exactly why the premium block-blob account in step 2 is worth it.

The net effect, as the numbers above show: cold-start time drops from download time + disk-load time (added together, ~187 s here) to roughly the streaming time alone (~10 s, overlapped), with the disk write-and-read-back eliminated entirely. That gap only widens as your models grow and your autoscaler gets busier: the larger the model weights, the more disk staging you skip, and the more often you scale out at peak traffic, the more those saved seconds compound into lower tail latency and fewer cold-start stalls for your AKS inference apps. Let's build the infrastructure to prove it.

Prerequisites​

• The Azure CLI (az), version 2.85.0 or later.
• kubectl and jq.
• envsubst (ships with GNU gettext) used to fill the ${...} placeholders in the Kubernetes manifests before applying them. On macOS, brew install gettext; on Debian/Ubuntu, apt-get install gettext-base.
• An Azure subscription with quota for an NVIDIA GPU VM (For example: Standard_NC24ads_A100_v4) in your target region.
• Logged in: az login and az account set --subscription <id>.

Configuration​

Everything below is driven by these variables. Modify them to match your environment and export them once in your shell and the rest of the commands are copy-paste.

# --- Azure / cluster ---
export AZURE_RESOURCE_GROUP="runai-demo"
export AZURE_REGION="<location with GPU VM SKU support>"
export CLUSTER_NAME="llm"
export ADMIN_USERNAME="azuser"

# --- GPU node pool ---
export NODE_POOL_NAME="gpunodes"
export NODE_POOL_VM_SIZE="<Standard_NC24ads_A100_v4 or other GPU VM size>"
export NODE_POOL_NODE_COUNT=1

# --- Storage ---
# Must be globally unique, 3-24 chars, lowercase letters and digits only.
export STORAGE_ACCOUNT_NAME="<provide unique name>"
export STORAGE_CONTAINER_NAME="models"

# --- Workload identity ---
export IDENTITY_NAME="vllm-model-reader"
export SERVICE_ACCOUNT_NAME="runai-streamer"
export NAMESPACE="default"

# --- Model / vLLM ---
export MODEL_NAME="microsoft/phi-4"
export VLLM_IMAGE="vllm/vllm-openai:v0.23.0"

1. Deploy the AKS cluster​

1a. Enable the managed GPU preview​

The fully managed GPU node-pool experience is a preview feature. Register it once per subscription. This is what lets us skip the NVIDIA GPU operator entirely, AKS takes over installing and maintaining the driver, the NVIDIA GPU device plugin, and the NVIDIA DCGM metrics exporter on every GPU node.

# The managed-GPU flag lives in the aks-preview extension (>= 19.0.0b29).
az extension add --name aks-preview --upgrade

# Register the feature flag.
az feature register \
    --namespace Microsoft.ContainerService \
    --name ManagedGPUExperiencePreview

Registration is asynchronous and can take a few minutes. Re-run the command below until it reports Registered before moving on, starting the node pool in step 1c while the feature is still Registering makes that command fail.

az feature show \
    --namespace Microsoft.ContainerService \
    --name ManagedGPUExperiencePreview \
    --query properties.state -o tsv

Once it prints Registered, propagate the feature to the resource provider:

az provider register --namespace Microsoft.ContainerService

1b. Create the resource group and cluster​

First, create the resource group:

az group create \
    --name "${AZURE_RESOURCE_GROUP}" \
    --location "${AZURE_REGION}"

Then create the cluster. It needs three things switched on for workload identity to work later: --enable-oidc-issuer (so Kubernetes service-account tokens can be validated by Microsoft Entra ID), --enable-workload-identity (installs the mutating webhook that injects federated-token env vars into labeled pods), and --enable-managed-identity. This takes a few minutes:

az aks create \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --name "${CLUSTER_NAME}" \
    --location "${AZURE_REGION}" \
    --node-count 1 \
    --enable-oidc-issuer \
    --enable-workload-identity \
    --enable-managed-identity \
    --generate-ssh-keys \
    --admin-username "${ADMIN_USERNAME}" \
    --os-sku Ubuntu

Finally, fetch the cluster credentials so kubectl targets it:

az aks get-credentials \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --name "${CLUSTER_NAME}" \
    --overwrite-existing

The default node pool created here is a small CPU pool that runs system components. GPU capacity comes from a dedicated pool in the next step.

1c. Add the managed GPU node pool​

This single flag, --enable-managed-gpu=true, is the whole reason we don't need a GPU operator. AKS provisions the node with the NVIDIA driver already installed, deploys the NVIDIA GPU device plugin so nvidia.com/gpu is advertised, and wires up the NVIDIA DCGM exporter for metrics.

az aks nodepool add \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --cluster-name "${CLUSTER_NAME}" \
    --name "${NODE_POOL_NAME}" \
    --node-count "${NODE_POOL_NODE_COUNT}" \
    --node-vm-size "${NODE_POOL_VM_SIZE}" \
    --enable-managed-gpu=true

Confirm the GPU is schedulable before moving on. It can take a couple of minutes after the pool is ready for the device plugin to advertise the resource:

kubectl get nodes -o json \
    | jq -r '.items[] | {name: .metadata.name, gpu: .status.allocatable["nvidia.com/gpu"]}'

You're looking for a node reporting "gpu": "1".

2. Create a premium block-blob storage account​

Premium block blob (Premium_LRS + BlockBlobStorage) gives the streamer the high, consistent throughput that makes this whole approach worthwhile. A standard account works too, but you'll leave bandwidth on the table.

Create the storage account:

az storage account create \
    --name "${STORAGE_ACCOUNT_NAME}" \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --location "${AZURE_REGION}" \
    --sku Premium_LRS \
    --kind BlockBlobStorage

Then create the container that will hold the model weights:

az storage container create \
    --name "${STORAGE_CONTAINER_NAME}" \
    --account-name "${STORAGE_ACCOUNT_NAME}"

3. Set up workload identity for Blob access​

This is the part that lets pods talk to Blob with zero secrets. The chain looks like this:

We use Storage Blob Data Contributor (not just Reader) on purpose: the same identity is reused by both the upload Job (which needs to write) and the vLLM pod (which only reads).

Read the cluster's OIDC issuer URL, the trust anchor for the federated credential, and create the user-assigned managed identity that pods will run as:

# The cluster's OIDC issuer URL, the trust anchor for federated credentials.
export AKS_OIDC_ISSUER=$(az aks show \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --name "${CLUSTER_NAME}" \
    --query "oidcIssuerProfile.issuerUrl" -o tsv)

# Create a user-assigned managed identity.
az identity create \
    --name "${IDENTITY_NAME}" \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --location "${AZURE_REGION}"

Capture the identity's client ID and object ID, plus the storage account's resource ID, the next two commands reference them:

export IDENTITY_CLIENT_ID=$(az identity show \
    --name "${IDENTITY_NAME}" \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --query clientId -o tsv)

export IDENTITY_OBJECT_ID=$(az identity show \
    --name "${IDENTITY_NAME}" \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --query principalId -o tsv)

export STORAGE_ACCOUNT_ID=$(az storage account show \
    --name "${STORAGE_ACCOUNT_NAME}" \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --query id -o tsv)

Grant the identity read/write access on the storage account:

:::caution You need permission to create role assignments The az role assignment create step here writes to Microsoft.Authorization/roleAssignments, which requires an elevated role on the scope: Owner, User Access Administrator, or Role Based Access Control Administrator. Plain Contributor isn't enough. Without it you'll see:

(AuthorizationFailed) The client 'you@example.com' with object id '...' does not
have authorization to perform action 'Microsoft.Authorization/roleAssignments/write'
over scope '/subscriptions/.../storageAccounts/<account>/providers/Microsoft.Authorization/roleAssignments/...'

If you hit this, ask a subscription/resource-group administrator to either grant you one of those roles or run the role-assignment command for you. The rest of the setup is unaffected. :::

az role assignment create \
    --role "Storage Blob Data Contributor" \
    --assignee-object-id "${IDENTITY_OBJECT_ID}" \
    --assignee-principal-type ServicePrincipal \
    --scope "${STORAGE_ACCOUNT_ID}"

Finally, federate the Kubernetes service account to the managed identity, so a projected service-account token can be exchanged for a Microsoft Entra ID token:

az identity federated-credential create \
    --name "vllm-federated-cred" \
    --identity-name "${IDENTITY_NAME}" \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --issuer "${AKS_OIDC_ISSUER}" \
    --subject "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}" \
    --audience api://AzureADTokenExchange

Create a dedicated service account for these workloads, rather than reusing the namespace's shared default, so the workload-identity wiring stays off every other pod in the namespace. manifests/service-account.yaml declares it with two pieces of workload-identity metadata: an annotation (azure.workload.identity/client-id) that tells the webhook which managed identity to mint tokens for, and a label (azure.workload.identity/use: "true").

curl -sL https://raw.githubusercontent.com/Azure-Samples/aks-samples/refs/heads/master/runai-model-streamer/manifests/service-account.yaml \
    | envsubst \
    | kubectl apply -f -

A subtlety worth understanding: the label is what actually triggers token injection, but the webhook keys off the label on the pod, not the service account. The label on the service account here is optional; the authoritative switch is the azure.workload.identity/use: "true" label in each pod template (you'll see it on both the Job and the Deployment in later sections). A pod that runs as this service account but is missing that label gets no token injected, and fails to authenticate to Blob with no obvious error. So from here on, every workload that needs Blob access carries azure.workload.identity/use: "true" on its pod template, which earns it AZURE_CLIENT_ID, AZURE_TENANT_ID, and a projected federated token automatically.

:::note Give the role assignment a couple of minutes to propagate Azure RBAC role assignments are eventually consistent. If the upload Job in the next step starts too soon, it fails with AuthorizationPermissionMismatch. If you hit that, wait a couple of minutes (sleep 120) and let the Job's backoffLimit retry. To re-run the Job from scratch, delete it first, a Job is immutable, so a second kubectl apply fails with AlreadyExists:

kubectl delete job upload-model --ignore-not-found

:::

4. Upload the model weights from a Kubernetes Job​

The Job does three things: pull microsoft/phi-4 from HuggingFace, install azcopy, and copy the files into Blob using workload identity (AZCOPY_AUTO_LOGIN_TYPE=WORKLOAD). Because it runs in-cluster, the HuggingFace → Blob transfer rides Azure's network, not yours.

A few details worth calling out:

• The pod template carries the azure.workload.identity/use: "true" label, so azcopy finds the federated token automatically.
• --overwrite=ifSourceNewer makes retries cheap: a re-run skips blobs that already uploaded successfully instead of re-sending all the model weights.
• --exclude-path '.cache' skips HuggingFace's local cache directory.

The full manifest lives in Azure-Samples/aks-samples. Pull it, fill in the placeholders with envsubst, and apply, all in one copy-pastable command:

curl -sL https://raw.githubusercontent.com/Azure-Samples/aks-samples/refs/heads/master/runai-model-streamer/manifests/upload-model-job.yaml \
    | envsubst '${NAMESPACE} ${SERVICE_ACCOUNT_NAME} ${STORAGE_ACCOUNT_NAME} ${STORAGE_CONTAINER_NAME} ${MODEL_NAME}' \
    | kubectl apply -f -

The envsubst allow-list substitutes only those five variables, so the runtime shell variables inside the container script ($AZURE_STORAGE_ACCOUNT_NAME, the retry counters) are left untouched.

Wait for it to finish and check the logs:

kubectl wait --for=condition=complete job/upload-model --timeout=1800s

:::note A note on scaling to larger models The 1800s (30 minute) timeout is plenty for a model like microsoft/phi-4 (a 14.7B-parameter model, ~29 GB on disk in bf16). If you reuse this manifest for a 70B+ parameter model, two limits need raising together: bump this --timeout so the command doesn't give up while the Job is still uploading, and raise the ephemeral-storage request/limit on the Job, larger model weights that overflow the reserved scratch space get evicted with DiskPressure long before any timeout fires. :::

In a separate terminal you can follow the logs:

kubectl logs -f job/upload-model

Verify the blobs landed:

az storage blob list \
    --account-name "${STORAGE_ACCOUNT_NAME}" \
    --container-name "${STORAGE_CONTAINER_NAME}" \
    --prefix "${MODEL_NAME}" \
    --output table

5. Deploy vLLM streaming from Azure Blob​

Now the payoff. Two things turn on Blob streaming:

• --load-format runai_streamer selects the RunAI Model Streamer loader.
• --model az://${STORAGE_CONTAINER_NAME}/${MODEL_NAME} points at the blobs using the native az:// scheme. The AZURE_STORAGE_ACCOUNT_NAME env var tells the streamer which account to read from, and the pod's workload-identity label handles auth, no keys, no connection strings.

:::tip Tuning the streamer The RunAI Model Streamer exposes tunable parameters that you pass to vLLM as a JSON object via --model-loader-extra-config. A few worth knowing:

• '{"distributed":true}' enables distributed streaming across tensor-parallel ranks, useful once you scale --tensor-parallel-size past 1.
• '{"concurrency":16}' sets how many OS threads read tensors from storage into the CPU buffer; raise it to push more bandwidth against a premium account.
• '{"memory_limit":5368709120}' caps the CPU staging buffer (in bytes).

To set one, add the flag to the container's args list in the Deployment manifest, for example:

        - --model-loader-extra-config
        - '{"concurrency":16}'

:::

The pod mounts an in-memory emptyDir at /dev/shm because vLLM uses shared memory for inter-process communication. A startupProbe and readinessProbe on vLLM's /health endpoint keep the pod out of the Service until the model has finished streaming and the server is actually answering.

The full manifest, a Deployment plus a Service, is in Azure-Samples/aks-samples. It has no runtime-only variables, so envsubst can substitute everything:

curl -sL https://raw.githubusercontent.com/Azure-Samples/aks-samples/refs/heads/master/runai-model-streamer/manifests/vllm-deployment.yaml \
    | envsubst \
    | kubectl apply -f -

Wait for the rollout to complete:

kubectl rollout status deployment/phi-4 --timeout=600s

In a separate terminal you can follow the logs:

kubectl logs -f deployment/phi-4

How do you know the weights actually came from Blob through the RunAI streamer and not vLLM's default loader? Look for the streamer's own progress line, Loading safetensors using Runai Model Streamer. Its presence is the confirmation, the runai_streamer loader engaged and pulled the tensors from az:// directly into GPU memory:

...
(EngineCore pid=157) INFO 06-29 20:07:33 [gpu_model_runner.py:5092] Starting to load model /root/.cache/vllm/assets/model_streamer/4d80bd96...
Loading safetensors using Runai Model Streamer:   0% Completed | 0/243 [00:00<?, ?it/s]
Loading safetensors using Runai Model Streamer:  47% Completed | 114/243 [00:06<00:07, 17.13it/s]
Loading safetensors using Runai Model Streamer: 100% Completed | 243/243 [00:08<00:00, 27.80it/s]
(EngineCore pid=157) INFO 06-29 20:07:43 [gpu_model_runner.py:5187] Model loading took 27.39 GiB memory and 9.956121 seconds
...

The closing Model loading took ... seconds line reports how long the stream-and-load took end to end, a handy number to compare against the default download-then-load path.

6. Verify and test​

Once it's serving, port-forward and send a chat completion. The port-forward needs a moment to establish, so wait for /health to answer before firing the request:

kubectl port-forward svc/phi-4 8000:8000 &

curl -s http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "microsoft/phi-4",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 50
    }' | jq .

A JSON response with a choices[0].message.content field means the model loaded from Blob and is serving.

How it fits together​

The pieces that make this clean:

• Native az:// support in vLLM + runai-model-streamer means a stock vllm/vllm-openai image just works. No custom builds, no monkey-patching.
• Workload identity end to end. Neither the uploader nor the server ever sees a storage key, both authenticate with short-lived federated tokens, and the same managed identity does double duty (write for the Job, read for the server).
• Managed GPU node pool. --enable-managed-gpu=true hands the driver, device plugin, and DCGM exporter lifecycle to AKS, so there's no GPU operator to install, reconcile, or debug.
• Upload from inside Azure. Moving the HuggingFace → Blob copy into a Job trades your home uplink for Azure's backbone, turning an hours-long upload into a minutes-long one.

Trade-offs and downsides​

Streaming from Blob isn't the right fit for every model. Weigh these before adopting it:

• Every cold start re-streams the full weights. The streamer's advantage is overlapping the network and GPU copies, which pays off most for large models on the autoscaling critical path. For a small model that already loads quickly from local disk, that win is marginal, the simpler default download path (optionally with the weights baked into your image) may be the better trade. Measure cold-start time both ways for your model before committing.
• One upload per model, up front. Every new model, and every new version of a model, has to be pushed to Blob with the upload Job before it can be served. There's no way around this with this pattern: the streamer reads from Blob, so the weights have to land in Blob first.
• Safetensors only. The RunAI Model Streamer loads weights in Safetensors format. A model distributed only as PyTorch pickle (.bin / .pt) or GGUF has to be converted first, or it won't load through runai_streamer. microsoft/phi-4 ships as Safetensors, so the walkthrough above works as-is.
• You now own a second copy of the weights. The model lives in your premium Blob account in addition to its upstream source on HuggingFace. That's an ongoing storage cost, premium block blob isn't cheap at tens of gigabytes per model, and it's on you to re-run the upload when the upstream model changes, or your served copy quietly goes stale.

Conclusion​

Cold-start time is the tax you pay every time an inference replica scales up, and the default download-then-load path pays it twice, once to disk, once to the GPU. Streaming weights from Azure Blob with the RunAI Model Streamer collapses that into a single overlapped copy, and now that vLLM and runai-model-streamer speak az:// natively, it takes nothing more than --load-format runai_streamer, one environment variable, and a workload-identity binding on a stock vllm/vllm-openai image.

The pieces reinforce each other: a managed GPU node pool removes the operator toil, workload identity removes the secrets, and an in-cluster upload Job moves the weights over Azure's backbone instead of your uplink. The result is faster, cheaper cold starts when you're autoscaling, exactly when it matters most.

If you're serving large models on AKS and every cold start sits on the critical path, this is worth adopting. If your model is small and already loads quickly from disk, measure both paths first, the simpler default may still win. Either way, the building blocks here are reusable: swap in your own model, point the Job at it once, and let the streamer do the rest.

Cleanup​

The GPU VM is the expensive part, delete the resource group when you're done to stop the meter on everything in one shot:

az group delete --name "${AZURE_RESOURCE_GROUP}" --yes --no-wait

References​

• Azure-Samples/aks-samples: runai-model-streamer scripts and manifests
• vLLM #34614: Azure Blob support in the RunAI streamer loader
• runai-model-streamer #116: az:// scheme support
• AKS-managed GPU node pools (preview)
• Azure Workload Identity on AKS
• vLLM RunAI Model Streamer docs

This post wires up those three decisions on AKS as a single OpenAI-compatible endpoint: a trivial agent step lands on a cheap self-hosted model (KAITO + vLLM, placed by GPU state) while a hard step escalates to a frontier model on Azure OpenAI — every call authenticated, metered, and traced. We lean on managed Azure throughout (KAITO for serving, Azure OpenAI for the frontier path, Azure Managed Prometheus and Grafana for observability); the two layers without a drop-in managed service — the semantic router and the AI gateway — run as open source on the cluster.

The three decisions​

"Route this request" hides three separate questions, each reading a different signal, each with a different owner. For an agent firing calls in a loop, the first two fire on every call and the third on every weak-path call:

1. Which model should answer this call? A multi-step reasoning hop deserves a frontier model; "extract the date from this string" does not. Answering means reading the request content and predicting answer quality. This is semantic routing, and we'll use RouteLLM.
2. How do I expose, secure, govern, and forward this traffic? Provider abstraction, auth, per-agent budgets, rate limits, guardrails, observability. None of that cares about prompt meaning — it's a policy job for a data plane. We'll use agentgateway, an OpenAI-compatible, agent-native proxy, in the cluster.
3. Which replica of the chosen model gets this call? Across a fleet of GPU pods the right target depends on KV-cache occupancy, queue depth, and which LoRA adapters are already resident — live GPU state. That's the Gateway API Inference Extension and its Endpoint Picker.

Content → policy → GPU state. That's the through-line, and the discipline is refusing to let any layer answer a question that belongs to another: the router never thinks about auth, the gateway never thinks about prompt difficulty, the Endpoint Picker never thinks about which model — only which replica.

The architecture on AKS​

Most slots are managed Azure; where an open-source piece has a managed alternative, it's called out in the layer that earns it:

With the map in hand, the rest of this post builds it — bottom-up, serving first and the semantic decision last, because each layer needs the address of the one beneath it. First, the cluster everything runs on.

Prerequisites​

• An Azure subscription with GPU quota for the VM family you'll serve on (this guide uses a single Standard_NC* GPU node pool, provisioned for you by KAITO). Quota isn't capacity — confirm the SKU is actually allocatable in your region before you rely on it.
• A recent Azure CLI and kubectl.
• An Azure OpenAI (or AI Foundry) resource with a current frontier deployment for the strong path. We use gpt-5.1; gpt-4o is deprecating in many regions and may not be newly deployable.

Create a cluster with the serving and monitoring add-ons enabled. KAITO provisions and manages GPU node pools on demand, so you don't pre-create them:

az aks create \
  --resource-group $RG \
  --name $CLUSTER \
  --node-count 2 \
  --enable-ai-toolchain-operator \
  --enable-oidc-issuer \
  --enable-azure-monitor-metrics \
  --generate-ssh-keys

az aks get-credentials --resource-group $RG --name $CLUSTER

That's the whole cluster — no gateway add-on, no extra CRDs, no Istio to bootstrap. agentgateway is the front door (more on that in Layer 2). With the cluster up, start at the bottom: serving the weak model.

Layer 3 — Serving, and the load-balancing trap​

Start with the failure mode, because it's the whole reason this layer is more than a Service — and agents make it worse.

Round-robin assumes requests are interchangeable. LLM requests are anything but: a 100k-token context-stuffed agent step and a 200-token tool-argument fill differ by three orders of magnitude in cost. An agent fleet mixes both constantly. Spray them uniformly across identical pods and round-robin will cheerfully queue a cheap completion behind a giant prefill on a saturated pod while an idle pod sits next to it. Your throughput looks fine and your p99 — the thing an agent loop feels on every iteration — falls off a cliff.

The fix isn't more pods — it's placing each request on the pod best able to take it right now, by reading live KV-cache occupancy and queue depth. That capability is what the Endpoint Picker is. Everything else in this layer exists to feed it.

3.1 The pods to place: KAITO​

The Endpoint Picker needs a homogeneous pool of model servers exporting the right signals, and KAITO produces exactly that: declare a Workspace, point it at a preset, and KAITO provisions the GPU SKU, runs vLLM, and exposes an OpenAI-compatible Service. Because the engine is vLLM, every pod emits the metrics the Endpoint Picker routes on (vllm:num_requests_waiting, vllm:kv_cache_usage_perc). Apply kaito-workspace.yaml — it pins instanceType: Standard_NC24ads_A100_v4, count: 2, and preset: phi-4-mini-instruct:

kubectl create namespace llm
kubectl apply -f kaito-workspace.yaml
kubectl get workspace workspace-phi-4-mini -n llm -w   # Ready can take ~10–20 min

KAITO provisions the GPU node, pulls the model, and starts vLLM; the workspace flips to Ready when inference is up:

NAME                   INSTANCE                   RESOURCEREADY   INFERENCEREADY   STATE
workspace-phi-4-mini   Standard_NC24ads_A100_v4   True            True             Ready

KAITO publishes a Service named workspace-phi-4-mini on port 80 and labels its pods kaito.sh/workspace: workspace-phi-4-mini — that label is how the next step selects the pool. One detail that will bite you later: the Service maps port 80 to the vLLM container's port 5000, and the Endpoint Picker routes to pod IPs directly, bypassing the Service. So the pool below targets 5000, not 80.

3.2 The placement logic: the Inference Extension​

Pool membership — which pods are the model server — is a different concern from how a workload is weighted against that pool, so the Inference Extension splits them across two objects:

• An InferencePool — the set of model-server pods plus a reference to the Endpoint Picker (EPP), the extension that an ext-proc-capable proxy calls to choose an endpoint per request. In this design that proxy is agentgateway (Layer 2); the InferencePool and EPP don't care who calls them.
• An InferenceObjective — the per-workload treatment: an integer priority against a pool. It replaces the older InferenceModel object: where InferenceModel carried a modelName and a criticality enum, InferenceObjective keeps only an integer priority and takes the served-model name from the request instead.

Install the CRDs and the Endpoint Picker (the project ships a Helm chart), pointing it at the KAITO pods. The EPP must serve plaintext gRPC to match agentgateway's ext-proc client — set --secure-serving=false, or the call to it resets before headers:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.0/manifests.yaml

helm install phi-epp \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool \
  --version v1.0.0 -n llm \
  --set inferencePool.modelServers.matchLabels."kaito\.sh/workspace"=workspace-phi-4-mini \
  --set provider.name=none \
  --set inferenceExtension.extraArgs="{--secure-serving=false}"

The Helm chart creates its own InferencePool named after the release, so you don't strictly need to apply one. If you do, inference-pool.yaml has the full InferencePool + InferenceObjective — with two fields that trip people up. The pool's selector needs a structured matchLabels, and its targetPorts must be the vLLM container port 5000 (the pool routes to pod IPs, bypassing the Service's port 80):

spec:
  selector:
    matchLabels: { kaito.sh/workspace: workspace-phi-4-mini }
  targetPorts:
    - number: 5000          # the vLLM container port, NOT the Service's 80

The other trap is the API group: InferencePool lives in inference.networking.k8s.io, but InferenceObjective lives in inference.networking.x-k8s.io — different groups, easy to miss.

The Endpoint Picker is now running and watching the pool, but nothing calls it yet. It needs an ext-proc-capable proxy out front to consult it per request — and that proxy is the AI gateway in Layer 2, so we wire the call there rather than standing up a separate Gateway API gateway just to reach it.

What this layer decided: only which replica, on GPU state. It has no idea what a "strong" model is, no budgets, no auth. Those are the next layer's job.

Layer 2 — The gateway, and why agents need it most​

The tempting shortcut is to let the router call backends directly. Resist it — and with agents the reason is sharper than with chat. An agent loop can spend an unbounded number of tokens before it decides to stop; without a budget and a rate limit somewhere on the path, a single misbehaving agent can run up a frontier bill or saturate your GPU fleet. A gateway is where that policy lives, written once, so nothing else has to reinvent it.

agentgateway is an open-source, OpenAI-compatible AI gateway built for exactly this traffic: provider abstraction, auth, token-based rate limiting, model-cost tracking, guardrails, and OpenTelemetry across backends — and, because it's agent-native, first-class MCP tool-serving and agent-to-agent routing when you grow into them. Here it does three jobs: hide the strong (Azure OpenAI) and weak (in-cluster) backends behind one endpoint, enforce policy on every call, and — because it speaks ext-proc itself — call the Endpoint Picker directly so the weak path gets GPU-state placement. That third job is what lets us drop the Gateway API gateway entirely: one data plane instead of two, and no dependency on the managed gateway's inference-extension support, which is still preview-gated on AKS today.

The outer shape is binds → listeners → routes → backends. The one thing to internalize: an AI backend is ai: { name, provider: { <provider>: {...} } } — the provider nests under provider:, name is required, and the Azure provider needs resourceName and resourceType (its model is your Azure deployment name). Auth attaches as a route policy whose key the Azure provider sends as the api-key header (a static key keeps the example self-contained; for production on AKS, prefer Azure workload identity, which agentgateway supports natively for the Azure provider):

# the strong route's backend (excerpt)
backends:
  - ai:
      name: aoai-strong
      provider:
        azure:
          resourceName: "<your-aoai-resource>"
          resourceType: openAI
          model: gpt-5.1
          apiVersion: "2024-10-21"

The strong route is a single ai backend pointed at Azure OpenAI. The weak route is where the front-door magic lives: a service backend pointed at the KAITO pods, with an inferenceRouting policy that hands each request to the Endpoint Picker over ext-proc and forwards to the pod it returns. Three details earned during validation, because the loader is strict about each:

# top-level: declare the model-server pool the picker selects from
services:
  - name: phi-weak
    namespace: llm
    hostname: workspace-phi-4-mini.llm.svc.cluster.local
    vips: []                       # required, even when empty
    ports: { 5000: 5000 }          # the vLLM container port

# ...the weak route's backend:
backends:
  - service:
      name: llm/workspace-phi-4-mini.llm.svc.cluster.local   # namespace/HOSTNAME, not the service name
      port: 5000
    policies:
      inferenceRouting:            # a BACKEND policy, not a route policy
        endpointPicker: { host: phi-epp-epp.llm.svc.cluster.local:9002 }
        destinationMode: passthrough   # trust the picker's choice

The full two-route config — including the strong ai backend and both routes' policies — is in agentgateway-config.yaml. The credential never leaves the gateway, and on the weak route agentgateway itself is the thing that consults the picker.

The governance that makes this safe for agents lives there too: the strong route carries a token rate limit (a localRateLimit route policy) — the backstop against an agent loop that won't stop spending. agentgateway can also price every call in real dollars through a top-level config.modelCatalog, but its schema is nested and provider-scoped — the obvious flat form didn't load for us — so wire it from the agentgateway docs for your release rather than copying a snippet.

Smoke-test both paths before the router exists:

# strong → Azure OpenAI
curl -s http://agentgateway.llm.svc.cluster.local:4000/strong/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.1","messages":[{"role":"user","content":"Say no in one word."}]}'

# weak → KAITO via the Endpoint Picker
# (same shape RouteLLM sends: it appends /v1/chat/completions to api_base)
curl -s http://agentgateway.llm.svc.cluster.local:4000/weak/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"phi-4-mini-instruct","messages":[{"role":"user","content":"Say hi in one word."}]}'

Both return an OpenAI-shaped completion — the strong call comes back stamped with the resolved Azure deployment, the weak call from the in-cluster model, and neither caller had to know where the model lived:

// strong → Azure OpenAI
{ "model": "gpt-5.1-2025-11-13",
  "choices": [{ "message": { "role": "assistant", "content": "No." } }],
  "usage": { "total_tokens": 29 } }

// weak → KAITO / vLLM
{ "model": "phi-4-mini-instruct",
  "choices": [{ "message": { "role": "assistant", "content": "Hello." } }] }

What this layer decided: how to authenticate, meter, guardrail, and forward — on policy, never on content. It dispatches whatever model name it's handed. Choosing that name is Layer 1's job.

Layer 1 — The semantic decision, and the threshold that is the whole ballgame​

Here's the only layer that reads the prompt — and the one decision that actually moves the bill. For an agent firing hundreds of calls per task, it's also the layer with the most leverage: every easy call it keeps off the frontier model is multiplied by the loop.

RouteLLM is a set of routers trained on human-preference data to predict, per prompt, whether a cheap model can match a strong model's answer. On the paper's own model pair, the featured mf (matrix-factorization) router reaches ~95% of GPT-4's quality on MT-Bench while making just ~26% of the GPT-4 calls — cutting cost by up to ~85% versus sending everything to GPT-4, by escalating only the genuinely hard prompts.

The composition is the elegant part: RouteLLM's strong and weak upstreams are just agentgateway. It makes the semantic call, picks a model, and forwards into the governed data plane — so the decision layer inherits all of Layer 2's auth, budgets, and tracing without re-implementing a line of it.

One wiring detail the reviewer in you should catch: agentgateway routes by path (/strong, /weak), but RouteLLM hands its choice to litellm as a model name. So point each model at the matching agentgateway path with a per-model api_base rather than a single base URL:

# inside the RouteLLM container — each model maps to its agentgateway route
python -m routellm.openai_server --routers mf \
  --strong-model openai/gpt-5.1 \
  --weak-model   openai/phi-4-mini-instruct \
  --config routellm-config.yaml   # sets api_base per model → /strong and /weak

One more gotcha from running this against current models: RouteLLM sends presence_penalty / frequency_penalty by default, and litellm (which RouteLLM uses to reach each provider) raises UnsupportedParamsError when the target model doesn't accept them — newer frontier models like the gpt-5.x family don't. Set litellm.drop_params=True so litellm silently drops the unsupported params instead of failing the call.

Clients call RouteLLM with a model string that encodes both the router and the cost threshold:

router-mf-0.11593
        │   └── threshold: the strong/weak cutoff — calibrate it
        └────── the matrix-factorization router

That threshold is the ballgame. Set it too low and you escalate everything and pay frontier prices for "hello"; set it too high and quality craters on the hard steps that needed the strong model. Don't guess it — calibrate against a representative sample so a target fraction of your traffic escalates:

python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5
# → 0.11593: the threshold that sends ~50% of calls to "strong"

Then close the loop with the real numbers. Run live agent traffic and read the actual strong/weak split off agentgateway's metrics — not RouteLLM's estimate — and, once you've wired the cost catalog, the dollars too; nudge the threshold until the cost/quality trade sits where you want it. The router proposes; your own traffic disposes.

One caveat the rate card can't see: prompt caching. A cached input token costs a fraction of an uncached one — Azure OpenAI discounts cache hits, and the weak path's vLLM prefix cache is the very thing the Endpoint Picker routes on. Both caches are per-model and per-prefix, so per-call routing works against them: bounce a session between models and each one's cache for that growing context goes cold, so the marginal cost of a strong call whose prefix is still warm is much lower than its sticker rate, and the savings from offloading it are smaller than the headline prices imply. This is another reason to measure billed cost rather than multiply tokens by a static rate — and, where caching dominates over difficulty, to route at session or task granularity instead of per call. Per-call routing on prompt difficulty is still the right default; just don't let a token-rate estimate convince you the split is cheaper or more expensive than your bill says.

Observability with managed Azure monitoring​

Layer 1 trades quality for cost, so this section closes one question: can you trust the routing numbers, and can you watch the same signals the Endpoint Picker routes on? The answer splits by source — GPU and latency come from vLLM; the strong/weak split (and, once you wire the cost catalog, dollars) comes from agentgateway — so a complete view scrapes both.

To get there, add two PodMonitors to the Managed Prometheus you enabled at cluster creation: one selecting the KAITO pods for vLLM's metrics on port 5000, one for agentgateway's on port 15020. Then link Azure Managed Grafana to the workspace. (One gotcha: the managed add-on uses Azure's CRDs under the azmonitoring.coreos.com API group, not upstream monitoring.coreos.com, so change a community manifest's apiVersion to azmonitoring.coreos.com/v1 or the add-on won't scrape it.)

The full path, one call​

# trivial step → router picks weak → agentgateway → Endpoint Picker → KAITO pod
curl -s http://routellm.llm.svc.cluster.local:6060/v1/chat/completions \
  -d '{"model":"router-mf-0.11593","messages":[{"role":"user","content":"What is 2+2?"}]}'

# hard step → router picks strong → agentgateway → Azure OpenAI
curl -s http://routellm.llm.svc.cluster.local:6060/v1/chat/completions \
  -d '{"model":"router-mf-0.11593",
       "messages":[{"role":"user","content":"Derive the EM algorithm and explain why each step cannot decrease the log-likelihood."}]}'

The model field in each response is the receipt — it tells you which way the router decided. Against the calibrated threshold 0.11593, "What is 2+2?" scores a strong-win-rate of 0.09 (below the line → weak) while the EM-algorithm prompt scores 0.29 (above → strong), so they come back from different models:

// "What is 2+2?"             → "model": "phi-4-mini-instruct"   (weak, a fraction of a cent)
// "Derive the EM algorithm…" → "model": "gpt-5.1-2025-11-13"    (strong, frontier pricing)

Both hit the same endpoint with the same model string. Three decisions — content, policy, GPU state — resolved in one round trip, each by the layer that owns it. Now point your agent framework's base URL at that endpoint and every call in the loop gets sorted the same way.

Scale it to your problem​

You rarely need all of it on day one. Because the layers don't bleed into each other, you can adopt exactly the slice your problem demands:

• One hosted model, a few agents, no self-hosting? Layer 2 only — agentgateway for auth, budgets, and guardrails over Azure OpenAI. There's nothing to route to and no GPUs to place on.
• Self-hosting at scale but one model class? KAITO + the Inference Extension give you managed serving and cache-aware placement. Skip semantic routing — there's only one model.
• A real cost problem from agents sending easy steps to expensive models? Add RouteLLM. It earns its keep the moment you have a meaningful price gap and a meaningful fraction of easy traffic — which describes essentially every agent in a loop.

Conclusion​

Routing agent traffic isn't one decision — it's three, each on its own signal: content picks the model (RouteLLM), policy governs the call (agentgateway), and GPU state picks the replica (the Inference Extension over KAITO-served vLLM). Keep those seams clean and you get a single OpenAI-compatible endpoint your agents point at, where every call is routed, metered, and traced by the layer that owns that concern — and where the easy calls an agent loop fires by the hundred quietly stay off the frontier bill.

From here, the obvious next moves all reinforce the same shape: autoscale the weak fleet on vllm:num_requests_waiting with the managed KEDA add-on so the GPU pool breathes with bursty agent traffic; lean into agentgateway's agent-native side by serving tools over MCP and routing agent-to-agent traffic through the same governed data plane; and add per-agent budgets at the gateway so spend stays attributable as more agents share the cluster.

The example manifests have everything here ready to adapt. For the components themselves, see KAITO on AKS, the Gateway API Inference Extension, agentgateway, RouteLLM (and its paper), and Managed Prometheus and Grafana for AKS.

The official quickstart walks you through deploying Anyscale on Azure using the Azure CLI and Azure Portal with Envoy Gateway for ingress. That's a solid starting point, but if you're like me, you want everything in Terraform so your infrastructure is repeatable, version-controlled, and easy to tear down.

In this post, we'll walk through a Terraform configuration that deploys the full Anyscale on Azure stack, swapping out Envoy Gateway for AKS managed Gateway API with Istio-based app routing. With this approach, AKS handles the gateway lifecycle and you skip the step of installing and configuring Envoy Gateway entirely.

What we're building​

Before we dive into the code, here's a quick overview of what the Terraform configuration creates:

• A resource group to hold everything
• An AKS cluster (Standard tier) with Gateway API and Istio-based app routing enabled
• The Anyscale operator installed as an AKS cluster extension
• A user-assigned managed identity with a federated credential for the Anyscale operator
• An Azure Container Registry (Standard SKU) for container images
• A storage account with hierarchical namespace (HNS) enabled and a private blob container
• The Anyscale Cloud and Cloud Resource registered via the AzAPI provider (Anyscale.Platform/clouds)
• Role assignments for storage, container registry, and AKS access

The full Terraform code is available as a GitHub Gist. In this post, I'll walk through each piece so you understand what's happening and why.

Prerequisites​

Before you get started, make sure you have the following:

• Azure CLI authenticated to a subscription
• Terraform
• Anyscale CLI (installed via pipx)
• jq for parsing Terraform outputs
• kubectl for interacting with the AKS cluster

Your subscription also needs the following resource providers registered:

• Anyscale.Platform
• Microsoft.ContainerService
• Microsoft.ContainerRegistry
• Microsoft.Storage
• Microsoft.ManagedIdentity

Step 1: Set up the Terraform providers​

Create a new directory for your Terraform configuration and create a file named main.tf. For simplicity, we'll put everything in a single file so it's easy to follow along. In a real project, you'd probably want to split things up into separate files. See the Terraform style guide on file names for best practices.

We'll need a few providers: azurerm for Azure resources, azapi for the Anyscale Platform resources, azuread for the service principal lookup, and a couple of utility providers. Add the following to your main.tf:

terraform {
  required_providers {
    azapi = {
      source  = "Azure/azapi"
      version = "~> 2.10.0"
    }

    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.8.0"
    }

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.76.0"
    }

    local = {
      source  = "hashicorp/local"
      version = "~> 2.9.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.9.0"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

We're using azapi here because the Anyscale.Platform resource type is brand new and not yet in the azurerm provider. The azapi provider lets us work with any ARM resource type directly, including preview ones. At the time of this writing, the Anyscale.Platform resource provider is not yet fully documented. I'll go into how I figured out the API schema for undocumented resources like this when we get to Step 5.

We also set prevent_deletion_if_contains_resources = false on the azurerm provider so that terraform destroy can cleanly remove the resource group without getting blocked by the resources inside it.

Step 2: Create the foundational resources​

Let's start with the basics: a resource group, some random naming helpers, and the storage and container registry resources that Anyscale needs. Add the following to your main.tf:

variable "location" {
  description = "Azure region. Must support Anyscale on Azure."
  type        = string
  default     = "westus3"
  validation {
    condition     = contains(["westcentralus", "eastus", "eastus2", "westus2", "westus3", "southcentralus"], var.location)
    error_message = "Must be a region that supports Anyscale on Azure."
  }
}

data "azurerm_client_config" "current" {}

resource "random_integer" "example" {
  min = 10
  max = 99
}

resource "random_string" "example" {
  length  = 4
  special = false
  upper   = false
  lower   = true
  numeric = false
}

locals {
  random_name = "anyscale${random_integer.example.result}"
}

resource "azurerm_resource_group" "example" {
  name     = "rg-${local.random_name}"
  location = var.location
}

The location variable has a validation block that restricts you to regions where Anyscale on Azure is currently supported. This saves you from potential deployment failures later.

Next, add the container registry and storage account:

resource "azurerm_container_registry" "example" {
  name                = "${local.random_name}${random_string.example.result}"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "Standard"
  admin_enabled       = false

  tags = {
    "anyscale-cloud" = local.random_name
  }
}

resource "azurerm_storage_account" "example" {
  name                            = "${local.random_name}${random_string.example.result}"
  resource_group_name             = azurerm_resource_group.example.name
  location                        = azurerm_resource_group.example.location
  account_kind                    = "StorageV2"
  account_tier                    = "Standard"
  account_replication_type        = "LRS"
  access_tier                     = "Hot"
  min_tls_version                 = "TLS1_2"
  https_traffic_only_enabled      = true
  is_hns_enabled                  = true
  default_to_oauth_authentication = true
  shared_access_key_enabled       = true

  blob_properties {
    cors_rule {
      allowed_headers    = ["*"]
      allowed_methods    = ["DELETE", "GET", "HEAD", "POST", "PUT"]
      allowed_origins    = ["https://*.anyscale.com"]
      exposed_headers    = ["*"]
      max_age_in_seconds = 3600
    }
  }

  tags = {
    "anyscale-cloud" = local.random_name
  }
}

resource "azurerm_storage_container" "example" {
  name                  = "anyscale-data"
  storage_account_id    = azurerm_storage_account.example.id
  container_access_type = "private"
}

A few things to note here. The storage account has is_hns_enabled = true because Anyscale uses Azure Data Lake Storage Gen2 (ADLS Gen2) for its cloud storage. HNS gives you hierarchical file system semantics on top of blob storage, which is what the abfss:// URI scheme expects. The CORS rule allows the Anyscale web console to interact with the storage account directly from the browser.

Step 3: Create the AKS cluster with Gateway API​

This is where things get interesting. Instead of using the azurerm_kubernetes_cluster resource, we're using azapi_resource to create the AKS cluster. The managed Gateway API and Istio-based app routing features also went GA recently, but the ability to configure these features hasn't landed in the azurerm provider yet. So we use azapi to hit the AKS API directly. Add the following to your main.tf:

resource "azapi_resource" "aks" {
  type      = "Microsoft.ContainerService/managedClusters@2026-03-02-preview"
  parent_id = azurerm_resource_group.example.id
  location  = azurerm_resource_group.example.location
  name      = "aks-${local.random_name}"

  schema_validation_enabled = false

  body = {
    identity = {
      type = "SystemAssigned"
    },
    properties = {
      dnsPrefix = "aks-${local.random_name}"
      agentPoolProfiles = [
        {
          name              = "systempool"
          mode              = "System"
          enableAutoScaling = true
          minCount          = 3
          maxCount          = 6
        }
      ]
      ingressProfile = {
        gatewayAPI = {
          installation = "Standard"
        }
        webAppRouting = {
          gatewayAPIImplementations = {
            appRoutingIstio = {
              mode = "Enabled"
            }
          }
        }
      }
      oidcIssuerProfile = {
        enabled = true
      }
      securityProfile = {
        workloadIdentity = {
          enabled = true
        }
      }
    }
    sku = {
      name = "Base"
      tier = "Standard"
    }
  }

  response_export_values = [
    "properties.oidcIssuerProfile.issuerURL",
    "properties.identityProfile.kubeletidentity.objectId"
  ]
}

Let's break down the key parts of the ingressProfile:

• gatewayAPI.installation = "Standard" installs the Gateway API CRDs and controller on the cluster
• webAppRouting.gatewayAPIImplementations.appRoutingIstio.mode = "Enabled" enables the Istio-based implementation, which gives you the approuting-istio gateway class

We also enable OIDC issuer and workload identity on the cluster. These are required for the Anyscale operator to authenticate with Azure services using federated credentials instead of storing secrets.

The response_export_values block tells azapi to capture the OIDC issuer URL and the kubelet identity object ID from the cluster response, which we'll need for the federated credential and role assignments later.

Step 4: Set up identity and role assignments​

The Anyscale operator needs a managed identity with specific permissions to access resources on your behalf. We'll create a user-assigned managed identity, set up a federated credential so it can authenticate from within the cluster, and assign the necessary roles. Add the following to your main.tf:

resource "azuread_service_principal" "example" {
  client_id    = "086bc555-6989-4362-ba30-fded273e432b" # Anyscale Kubernetes Operator Auth
  use_existing = true
}

resource "azurerm_user_assigned_identity" "example" {
  location            = azurerm_resource_group.example.location
  name                = "${local.random_name}-operator-identity"
  resource_group_name = azurerm_resource_group.example.name

  tags = {
    "anyscale-cloud" = local.random_name
  }
}

resource "azurerm_federated_identity_credential" "example" {
  name                      = azurerm_user_assigned_identity.example.name
  audience                  = ["api://AzureADTokenExchange"]
  issuer                    = azapi_resource.aks.output.properties.oidcIssuerProfile.issuerURL
  user_assigned_identity_id = azurerm_user_assigned_identity.example.id
  subject                   = "system:serviceaccount:anyscale-operator:anyscale-operator"
}

The azuread_service_principal resource references the well-known Anyscale Kubernetes Operator Auth application in your tenant. The use_existing = true flag tells Terraform to look up the existing service principal rather than trying to create one. This is needed so the Anyscale operator can authenticate against your cluster.

The federated identity credential is the glue between the Kubernetes service account (anyscale-operator in the anyscale-operator namespace) and the Azure managed identity. When the operator pod starts, it can exchange its Kubernetes service account token for an Azure access token without needing any stored secrets.

Now add the role assignments:

resource "azurerm_role_assignment" "acr_pull" {
  principal_id                     = azapi_resource.aks.output.properties.identityProfile.kubeletidentity.objectId
  role_definition_name             = "AcrPull"
  scope                            = azurerm_container_registry.example.id
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "blob_data_owner" {
  principal_id                     = azurerm_user_assigned_identity.example.principal_id
  role_definition_name             = "Storage Blob Data Owner"
  scope                            = azurerm_storage_account.example.id
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "acr_push" {
  principal_id                     = azurerm_user_assigned_identity.example.principal_id
  role_definition_name             = "AcrPush"
  scope                            = azurerm_container_registry.example.id
  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "acr_task_contributor" {
  principal_id                     = azurerm_user_assigned_identity.example.principal_id
  role_definition_name             = "Container Registry Tasks Contributor"
  scope                            = azurerm_container_registry.example.id
  skip_service_principal_aad_check = true
}

Here's what each role does:

• AcrPull for the kubelet identity so the cluster can pull container images
• Storage Blob Data Owner for the operator identity to read/write data in the storage account
• AcrPush and Container Registry Tasks Contributor for the operator identity to push images and run ACR tasks

Step 5: Register the Anyscale Cloud resources​

Now we register the Anyscale Cloud and Cloud Resource using the Anyscale.Platform resource provider through azapi. These resources tell the Anyscale control plane about your Azure environment.

Add the following to your main.tf:

resource "azapi_resource" "anyscale_cloud" {
  type      = "Anyscale.Platform/clouds@2026-02-01-preview"
  parent_id = azurerm_resource_group.example.id
  location  = azurerm_resource_group.example.location
  name      = local.random_name

  schema_validation_enabled = false

  body = {
    properties = {
      acrResourceId = azurerm_container_registry.example.id
    }
  }

  tags = {
    "anyscale-cloud" = local.random_name
  }

  response_export_values = [
    "properties.ssoUrl",
    "properties.cloudResourceId"
  ]

  depends_on = [
    azapi_resource.aks
  ]
}

resource "azapi_resource" "anyscale_cloud_resource" {
  type      = "Anyscale.Platform/clouds/cloudResources@2026-02-01-preview"
  parent_id = azapi_resource.anyscale_cloud.id
  location  = azurerm_resource_group.example.location
  name      = "default"

  schema_validation_enabled = false

  body = {
    properties = {
      provider                    = "Azure"
      computeStack                = "K8S"
      cloudStorageBucketEndpoint  = azurerm_storage_account.example.primary_blob_endpoint
      cloudStorageBucketName      = "abfss://${azurerm_storage_container.example.name}@${azurerm_storage_account.example.primary_dfs_host}"
      anyscaleOperatorIamIdentity = azurerm_user_assigned_identity.example.principal_id
    }
  }

  tags = {
    "anyscale-cloud" = local.random_name
  }

  response_export_values = [
    "properties.cloudResourceId"
  ]

  depends_on = [
    azapi_resource.aks
  ]
}

The Anyscale.Platform/clouds resource registers a new Anyscale "cloud" (their term for a connected infrastructure environment) and associates it with your container registry. The child cloudResources resource provides the details about compute (Kubernetes), storage (the ADLS Gen2 bucket), and the operator identity.

Notice the cloudStorageBucketName uses the abfss:// URI format. This is the Azure Blob File System (ABFS) scheme that ADLS Gen2 uses, and it's how Anyscale accesses your storage.

Step 6: Install the Anyscale operator extension​

The Anyscale operator is deployed as an AKS cluster extension. This is where we wire up the gateway configuration. Add the following to your main.tf:

resource "azurerm_kubernetes_cluster_extension" "anyscale_operator" {
  name           = "anyscaleoperator"
  cluster_id     = azapi_resource.aks.id
  extension_type = "Anyscale.AKS.Operator"
  release_train  = "stable"

  plan {
    name      = "anyscale-operator"
    product   = "anyscale-operator-aks"
    publisher = "anyscale1750870039553"
  }

  configuration_settings = {
    "global.auth.audience"          = "api://086bc555-6989-4362-ba30-fded273e432b/.default"
    "global.auth.iamIdentity"       = azurerm_user_assigned_identity.example.client_id
    "global.cloudDeploymentId"      = azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId
    "global.controlPlaneURL"        = "https://console.azure.anyscale.com"
    "workloads.serviceAccount.name" = "anyscale-operator"
    "networking.gateway.enabled"    = "true"
    "networking.gateway.name"       = "gateway"
    "networking.gateway.className"  = "approuting-istio"
    "networking.gateway.namespace"  = "anyscale-operator"
    "networking.gateway.apiVersion" = "gateway.networking.k8s.io/v1"
    "networking.gateway.hostname"   = "${replace(azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId, "_", "-")}.${azapi_resource.aks.location}.cloudapp.azure.com"
  }

  depends_on = [
    azurerm_federated_identity_credential.example
  ]
}

The key configuration settings to pay attention to:

• networking.gateway.className = "approuting-istio" tells the operator to use the AKS managed Istio gateway class
• networking.gateway.apiVersion = "gateway.networking.k8s.io/v1" ensures we're using the GA Gateway API spec
• networking.gateway.hostname constructs a DNS label that Azure will use to assign a public FQDN to the gateway's load balancer

The plan block references the Azure Marketplace listing for the Anyscale operator. This is how Azure knows what extension to install and which publisher it comes from.

We also need a role assignment so you can manage the Anyscale platform resources. Add this to your main.tf:

resource "azurerm_role_assignment" "anyscale_platform_contributor" {
  principal_id         = data.azurerm_client_config.current.object_id
  role_definition_name = "Anyscale Platform Contributor Role"
  scope                = azapi_resource.anyscale_cloud.id
}

Step 7: Outputs and the Gateway template​

Before we deploy, we need two more things: Terraform output values so we can grab resource names after the deployment, and a template file for the Gateway manifest.

First, add the local_file resource to your main.tf. It takes a template (which we'll create next) and renders it into anyscale-gateway.yaml with the actual cloud resource ID:

resource "local_file" "anyscale_gateway" {
  filename = "anyscale-gateway.yaml"
  content = templatefile("anyscale-gateway.tmpl",
    {
      ANYSCALE_CLOUD_RESOURCE_ID = replace(azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId, "_", "-")
    }
  )
}

Next, add the following outputs to your main.tf:

output "rg_name" {
  value = azurerm_resource_group.example.name
}

output "aks_name" {
  value = azapi_resource.aks.name
}

output "anyscale_cloud_id" {
  value = split("/", azapi_resource.anyscale_cloud.output.properties.ssoUrl)[4]
}

output "anyscale_cloud_resource_id" {
  value = azapi_resource.anyscale_cloud_resource.output.properties.cloudResourceId
}

Finally, create a file named anyscale-gateway.tmpl in the same directory. Terraform uses this template to generate the Gateway manifest with your cloud resource ID injected:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: anyscale-operator
spec:
  gatewayClassName: approuting-istio
  infrastructure:
    annotations:
      service.beta.kubernetes.io/azure-dns-label-name: ${ANYSCALE_CLOUD_RESOURCE_ID}
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Same
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "*.i.azure.anyscaleuserdata.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: anyscale-${ANYSCALE_CLOUD_RESOURCE_ID}-certificate
      allowedRoutes:
        namespaces:
          from: Same
    - name: https-session
      port: 443
      protocol: HTTPS
      hostname: "*.s.azure.anyscaleuserdata.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: anyscale-${ANYSCALE_CLOUD_RESOURCE_ID}-certificate
      allowedRoutes:
        namespaces:
          from: Same

The Gateway defines three listeners: one for HTTP traffic and two for HTTPS. The HTTPS listeners handle Anyscale's interactive (*.i.) and session (*.s.) subdomains with TLS termination. The azure-dns-label-name annotation tells Azure to assign a public DNS label to the gateway's load balancer IP.

Deploy it​

With all the Terraform code in place, it's time to deploy. Sign in to Azure and set your subscription:

az login
az account set -s <subscription-id>
export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)

Initialize Terraform to download the providers:

terraform init

Then format and validate to make sure everything is clean:

terraform fmt
terraform validate

If validation passes, go ahead and apply:

terraform apply

Review the plan and type yes to proceed. The deployment takes about 7-10 minutes since it's creating an AKS cluster, installing the Anyscale operator extension, and registering the cloud resources.

Once it's done, grab the outputs and connect to the cluster:

read -r \
  RG_NAME \
  AKS_NAME \
  ANYSCALE_CLOUD_ID \
  ANYSCALE_CLOUD_RESOURCE_ID <<< "$(terraform output -json | jq -r \
    '[.rg_name.value,
      .aks_name.value,
      .anyscale_cloud_id.value,
      .anyscale_cloud_resource_id.value] | @tsv')"

az aks get-credentials -g $RG_NAME -n $AKS_NAME

Then deploy the gateway:

kubectl apply -f anyscale-gateway.yaml

Verify the deployment​

Check that the Anyscale operator and gateway-related pods are running:

kubectl get po -A

You should see pods in the anyscale-operator namespace for the operator itself, and Istio-related pods in the gateway infrastructure namespaces.

Now sign in to the Anyscale console. I installed the Anyscale CLI using pipx, which I like because it installs Python CLI tools in isolated environments, similar to how npx works in the Node.js world. You get the tool without polluting your global Python packages.

pipx install anyscale
export ANYSCALE_HOST=https://console.azure.anyscale.com
anyscale login

Verify the cloud is registered and healthy:

anyscale cloud list
anyscale cloud verify --id $ANYSCALE_CLOUD_ID

The cloud list command should show your cloud with the AZURE provider and your region. The cloud verify command runs a series of checks against your cluster, including operator health, identity, and networking. You want to see all checks come back as PASSED.

From here, you can follow the quickstart guide to run your first workload. You can also open the Anyscale console directly from the Azure portal by navigating to your Anyscale resource and selecting the console link. That's where you can explore workspaces, submit jobs, and manage your Ray clusters.

Cleanup​

When you're done experimenting, tear everything down:

anyscale workspace_v2 terminate --project default -n <workspace-name>

terraform destroy

Review the plan and type yes to confirm. This will remove all the resources, including the AKS cluster, Anyscale cloud registration, storage, and container registry.

Summary​

In this post, we walked through deploying Anyscale on Azure using Terraform with AKS managed Gateway API and Istio-based app routing. Along the way, we picked up a useful technique for figuring out API schemas when the docs haven't caught up yet, and saw how the azapi provider lets you automate new Azure services before the azurerm provider adds support.

The full Terraform code is available as a GitHub Gist. Clone it, tweak the variables, and you should be up and running in under 10 minutes.

What's next?​

• Explore the Anyscale on Azure documentation for more configuration options
• Check out the Ray documentation for building distributed applications
• Try scaling your Ray cluster with different node pool configurations and GPU-enabled VMs
• Look into Anyscale workspaces for interactive development on your cluster

Have questions or run into issues? Drop us a note in GitHub Discussions or open an issue.

Happy deploying!

Resources​

• Full Terraform code for this walkthrough
• Anyscale on Azure public preview announcement
• What is Anyscale on Azure?
• Quickstart: Deploy Anyscale on Azure
• AKS managed Gateway API
• Istio-based app routing with Gateway API on AKS
• AzAPI Terraform provider
• Ray documentation

Since our March preview announcement, we've also shipped two of our most requested capabilities that gated a real production story for Gateway API on AKS:

• Azure DNS and Azure Key Vault, wired in for you. No more manually deploying a SecretProviderClass, a sync pod, or a separate external-dns instance just to get a TLS-terminated, DNS-resolvable hostname. Drop two tls.options on a listener, apply an ExternalDNS CR, and you're done.
• Access logs, enabled out of the box. Every gateway proxy writes a structured JSON access log line per request to stdout. kubectl logs the gateway deployment and you're already debugging — no Telemetry resource, no EnvoyFilter, no opt-in flag. This closes one of the last parity gaps with the nginx experience our users are accustomed to.

The rest of this post walks through both. We'll create a cluster, expose a sample app over the Gateway API, see access logs land on stdout, and then layer on DNS + TLS using the ClusterExternalDNS and ExternalDNS CRDs plus listener TLS options. A working DNS zone and Key Vault are part of the demo, so by the end you'll have HTTPS records resolving in a zone you own.

:::note Rollout

App Routing Istio and the Managed Gateway API installation both went GA as part of the AKS 20260428 release, which has rolled out to all regions — those features are available everywhere today. The DNS/TLS operator integration and access-logging-by-default ship with the AKS 20260529 release; track region-by-region progress on the AKS release tracker. Even following the release rolling out to your region, there may be some delays (on the order of a couple of days) that may prevent you from using it. In this case, it's best to use a region that AKS rolled out to earlier.

:::

What's new​

Access logs by default​

Every Envoy gateway proxy that AKS provisions for your Gateway resources now writes one structured JSON access log line per request to its container log out of the box. The very first thing you can do once traffic flows is kubectl logs the gateway deployment and watch requests arrive. No Istio resources needed to manage it, and no opt-in flag. This brings the Gateway API experience closer to parity with the nginx-based experience our users are accustomed to.

Because the logs land on stdout, they flow through the standard container log pipeline, so you can collect them centrally instead of tailing each gateway by hand. Enable the Azure Monitor (Container Insights) add-on, and each gateway proxy's access logs are collected into a Log Analytics workspace, where you can query, retain, and alert on them with KQL. You can enable it when you create or update the cluster (az aks create/update --enable-addons monitoring --workspace-resource-id <workspace-id>), or onboard an existing cluster through the portal — see Enable monitoring for Kubernetes clusters and the Monitor AKS overview. Once it's enabled, the per-request access log lines show up in the ContainerLogV2 table, so a query like ContainerLogV2 | where PodName startswith "httpbin-gateway-approuting-istio" gives you every request the gateway served across all replicas.

DNS and TLS integration via the App Routing operator​

At preview, DNS and TLS automation was a major piece of the ingress-nginx experience the Gateway API path didn't match. That gap is now closed via two paths in the App Routing operator:

• TLS — A pair of listener tls.options keys (kubernetes.azure.com/tls-cert-keyvault-uri and kubernetes.azure.com/tls-cert-service-account) tell the operator to provision a SecretProviderClass, sync the certificate from Azure Key Vault as a kubernetes.io/tls Secret, and patch the listener's certificateRefs for you.
• DNS — Two CRDs, ClusterExternalDNS (cluster-scoped) and ExternalDNS (namespace-scoped), let you stand up an external-dns instance scoped to one or more Azure DNS zones and tell it to watch Gateway/HTTPRoute/GRPCRoute resources. Records appear automatically based on the hostnames on your routes.

Demo: from cluster create to HTTPS-with-DNS​

The rest of this post is a hands-on walkthrough that takes you through our newly-GA'd features along with the enhancements we're so excited about. To follow along you'll need:

• Azure CLI 2.86.0 or later (az --version to check, az upgrade to update).
• An Azure subscription and resource group, plus permission to create resources in it.
• Sufficient Key Vault RBAC on your own Azure identity to create certificates — Key Vault Certificates Officer is the minimum, Key Vault Administrator is fine. Grant this on the vault you'll create in step 5b before running the cert-creation command.

Export these once and reuse them throughout:

export RESOURCE_GROUP=<your-resource-group>
export CLUSTER=<your-cluster-name>
export LOCATION=<your-azure-region>

1. Create the cluster (or update an existing one)​

The three flags that matter are --enable-app-routing-istio (turns on the App Routing Gateway API implementation), --enable-gateway-api (installs the AKS-managed Gateway API CRDs), and --enable-app-routing (deploys the App Routing operator, which manages the DNS/TLS integration). All APIs are GA — no preview-feature registration and no aks-preview extension required.

On a new cluster:

az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER \
  --location $LOCATION \
  --enable-app-routing-istio \
  --enable-app-routing \
  --enable-gateway-api \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --enable-addons azure-keyvault-secrets-provider

On an existing cluster:

az aks update \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER \
  --enable-app-routing \
  --enable-app-routing-istio \
  --enable-gateway-api \
  --enable-oidc-issuer \
  --enable-workload-identity

# CSI addon needs its own command on existing clusters
az aks enable-addons \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER \
  --addons azure-keyvault-secrets-provider

--enable-oidc-issuer and --enable-workload-identity are required for the DNS/TLS demo later — both DNS and TLS use Microsoft Entra Workload Identity to authenticate to Azure DNS and Key Vault. The azure-keyvault-secrets-provider addon installs the Secrets Store CSI Driver, which the operator uses to materialize Key Vault certificates as Kubernetes Secrets. Set them all now even if you don't have a workload to expose yet.

Pull credentials so kubectl works:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER

2. Confirm the control plane is up​

The App Routing Gateway API implementation runs istiod in the aks-istio-system namespace.

kubectl get pods -n aks-istio-system

NAME                      READY   STATUS    RESTARTS   AGE
istiod-54b4ff45cf-htph8   1/1     Running   0          3m15s
istiod-54b4ff45cf-wlvgd   1/1     Running   0          3m

You'll also see the app-routing-system namespace with NGINX still running:

kubectl get pods -n app-routing-system

You can ignore this for the rest of the demo. NGINX-by-default is on its way out as part of the Ingress-NGINX retirement. Once we stop deploying it after November 2026, it won't appear for users by default.

The approuting-istio GatewayClass is what you'll target with your Gateway resources:

kubectl get gatewayclass approuting-istio

NAME               CONTROLLER                    ACCEPTED   AGE
approuting-istio   istio.aks.azure.com/gateway-controller   True       4m

3. Expose a sample app via Gateway API​

We'll deploy httpbin, front it with a single Gateway listener over plain HTTP, attach an HTTPRoute, and verify traffic flows. (We'll make this HTTPS in section 5.)

Deploy httpbin in the default namespace:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/httpbin/httpbin.yaml

Create the Gateway and HTTPRoute:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  gatewayClassName: approuting-istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - name: httpbin-gateway
  hostnames: ["httpbin.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF

AKS provisions the Envoy Deployment, Service, HorizontalPodAutoscaler, and PodDisruptionBudget for the Gateway automatically. Wait for the Gateway to be programmed and grab its external IP:

kubectl wait --for=condition=programmed gateways.gateway.networking.k8s.io httpbin-gateway --timeout=120s
export INGRESS_HOST=$(kubectl get gateways.gateway.networking.k8s.io httpbin-gateway -ojsonpath='{.status.addresses[0].value}')

Send a request:

curl -s -I -H "Host: httpbin.example.com" "http://$INGRESS_HOST/get"

You should see an HTTP/1.1 200 OK.

4. See access logs on the gateway proxy​

Each Gateway resource is backed by a managed Deployment named <gateway-name>-approuting-istio:

kubectl get deployment httpbin-gateway-approuting-istio

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
httpbin-gateway-approuting-istio   2/2     2            2           1m

Tail its logs and re-issue a curl:

kubectl logs deploy/httpbin-gateway-approuting-istio --tail=20 -f

You should see an access log for each of your requests. That's the JSON-formatted Envoy access log written to stdout, on by default — useful for sanity checks, post-mortem analysis, or shipping into Azure Monitor / Log Analytics via the standard container log pipeline.

5. Add Azure DNS and TLS​

Now for the best part (well, at least in my opinion). We'll create an Azure DNS zone we own, an Azure Key Vault with a real cert for a hostname in that zone, a user-assigned managed identity that can write to both, and then wire everything in via two CRDs and two listener tls.options.

For this demo we'll use the zone ga-demo.dev with two sub-hosts, a.ga-demo.dev and b.ga-demo.dev. Substitute your own zone name.

export ZONE_NAME=ga-demo.dev
export KV_NAME=approuting-ga-demo-kv  # must be globally unique
export UAMI_NAME=approuting-ga-demo-id
export SA_NAME=approuting-demo-sa

5a. Create an Azure DNS zone​

az network dns zone create --resource-group $RESOURCE_GROUP --name $ZONE_NAME
export ZONE_ID=$(az network dns zone show --resource-group $RESOURCE_GROUP --name $ZONE_NAME --query id -o tsv)

The zone's authoritative nameservers are listed under az network dns zone show ... --query nameServers. You can find this on the Azure portal too. If you're using a registrar-owned domain in production, you'd point the registrar at those NS records. But for a one-off demo, you can resolve through the zone NS directly with dig @<ns> a.${ZONE_NAME} which is what we'll do later to ensure our curl resolves correctly.

5b. Create an Azure Key Vault and a self-signed wildcard cert​

We'll mint a self-signed wildcard cert directly in Key Vault so it covers both a.ga-demo.dev and b.ga-demo.dev from a single certificate. (For production, import a CA-issued PFX with az keyvault certificate import instead.)

az keyvault create \
  --name $KV_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --enable-rbac-authorization true

# Cert policy: wildcard subject + SAN so one cert covers every sub-host of the zone
cat > /tmp/cert-policy.json <<EOF
{
  "issuerParameters": { "name": "Self" },
  "keyProperties": { "exportable": true, "keyType": "RSA", "keySize": 2048, "reuseKey": false },
  "secretProperties": { "contentType": "application/x-pkcs12" },
  "x509CertificateProperties": {
    "subject": "CN=*.${ZONE_NAME}",
    "subjectAlternativeNames": { "dnsNames": ["*.${ZONE_NAME}", "${ZONE_NAME}"] },
    "validityInMonths": 12,
    "keyUsage": ["digitalSignature", "keyEncipherment"]
  }
}
EOF

az keyvault certificate create \
  --vault-name $KV_NAME \
  --name approuting-demo-cert \
  --policy @/tmp/cert-policy.json

# Capture the unversioned cert URI so the listener tracks rotations automatically
export CERT_URI=$(az keyvault certificate show \
  --vault-name $KV_NAME \
  --name approuting-demo-cert \
  --query id -o tsv | sed 's|/[^/]*$||')
echo "Cert URI: $CERT_URI"

5c. Create a managed identity, role assignments, and federated credentials​

Both the operator's DNS controller and the gateway listener's TLS sync authenticate to Azure as a user-assigned managed identity (UAMI), federated to a Kubernetes ServiceAccount via the cluster's OIDC issuer.

# 1. Create the UAMI
az identity create --resource-group $RESOURCE_GROUP --name $UAMI_NAME
export UAMI_CLIENT_ID=$(az identity show --resource-group $RESOURCE_GROUP --name $UAMI_NAME --query clientId -o tsv)
export UAMI_PRINCIPAL_ID=$(az identity show --resource-group $RESOURCE_GROUP --name $UAMI_NAME --query principalId -o tsv)

# 2. Grant DNS Zone Contributor on the zone, Key Vault Secrets User on the vault
az role assignment create \
  --assignee-object-id $UAMI_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "DNS Zone Contributor" \
  --scope $ZONE_ID

az role assignment create \
  --assignee-object-id $UAMI_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope $(az keyvault show --name $KV_NAME --query id -o tsv)

export OIDC_ISSUER=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER --query oidcIssuerProfile.issuerUrl -o tsv)

We'll create the app-a and app-b namespaces now and federate the same UAMI to a $SA_NAME ServiceAccount in each. Federated credentials are subject-specific — each (namespace, ServiceAccount) pair needs its own:

for ns in app-a app-b; do
  kubectl create namespace $ns

  az identity federated-credential create \
    --identity-name $UAMI_NAME \
    --resource-group $RESOURCE_GROUP \
    --name approuting-demo-fic-$ns \
    --issuer $OIDC_ISSUER \
    --subject "system:serviceaccount:$ns:$SA_NAME" \
    --audiences "api://AzureADTokenExchange"

  kubectl apply -n $ns -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: $SA_NAME
  annotations:
    azure.workload.identity/client-id: $UAMI_CLIENT_ID
  labels:
    azure.workload.identity/use: "true"
EOF
done

The azure.workload.identity/use: "true" label is required; it's what tells the AKS Workload Identity webhook to inject the projected OIDC token volume and the AZURE_* env vars into pods that consume this SA. Without it, the operator's placeholder pod gets a default Kubernetes SA token instead of one signed by your cluster's OIDC issuer, and Entra rejects it with AADSTS700211: No matching federated identity record.

The same ServiceAccount drives both the DNS controller and the listener's TLS sync. One identity, two roles in Azure, one SA per namespace.

5d. Deploy two gateways in two namespaces​

Deploy httpbin in each namespace, then create one Gateway per namespace with an HTTPS listener pointing at the Key Vault cert URI and the SA. Each gateway covers its own sub-host:

for ns in app-a app-b; do
  kubectl apply -n $ns -f https://raw.githubusercontent.com/istio/istio/release-1.27/samples/httpbin/httpbin.yaml
done

for pair in "app-a:a" "app-b:b"; do
  ns=${pair%%:*}; sub=${pair##*:}
  fqdn=${sub}.${ZONE_NAME}
  kubectl apply -n $ns -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: ${sub}-gateway
  labels:
    app: approuting-demo
    zone: ${sub}
spec:
  gatewayClassName: approuting-istio
  listeners:
  - name: https
    hostname: $fqdn
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        kubernetes.azure.com/tls-cert-keyvault-uri: $CERT_URI
        kubernetes.azure.com/tls-cert-service-account: $SA_NAME
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ${sub}-route
spec:
  parentRefs:
  - name: ${sub}-gateway
  hostnames: ["$fqdn"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF
done

The TLS options tell the App Routing operator to:

1. Resolve $CERT_URI against the Key Vault, using the workload identity bound to $SA_NAME in the listener's namespace.
2. Provision a SecretProviderClass named kv-gw-cert-<gateway>-<listener>.
3. Sync the certificate as a kubernetes.io/tls Secret of the same name in the listener's namespace.
4. Patch the listener's certificateRefs to point at the Secret.

You can confirm the SPCs and Secrets exist:

kubectl get secretproviderclass,secret -n app-a
kubectl get secretproviderclass,secret -n app-b

5e. Scenario A — ClusterExternalDNS: records for both gateways​

ClusterExternalDNS is cluster-scoped: a single CR points at one or more Azure DNS zones and watches Gateway resources cluster-wide. The operator deploys an external-dns instance in the namespace named under resourceNamespace; the SA in that namespace authenticates to Azure DNS.

Apply it:

kubectl apply -f - <<EOF
apiVersion: approuting.kubernetes.azure.com/v1alpha1
kind: ClusterExternalDNS
metadata:
  name: demo-cluster-dns
spec:
  resourceName: demo-cluster-dns
  resourceNamespace: app-a
  dnsZoneResourceIDs:
  - $ZONE_ID
  resourceTypes:
  - gateway
  identity:
    type: workloadIdentity
    serviceAccount: $SA_NAME
EOF

After a minute, both records show up in the zone:

az network dns record-set a list --resource-group $RESOURCE_GROUP --zone-name $ZONE_NAME -o table

Name    ResourceGroup              Ttl    Type    AutoRegistered    Metadata
------  -------------------------  -----  ------  ----------------  --------
a       <your-rg>                  300    A       False
b       <your-rg>                  300    A       False

That's ClusterExternalDNS: one CR, one external-dns deployment, records for both gateways across both namespaces.

5f. Scenario B — ExternalDNS: namespace-scoped, with label filtering​

Now flip the model. Delete the ClusterExternalDNS:

kubectl delete clusterexternaldns demo-cluster-dns

To make the namespace + label scope concrete, deploy a third gateway in app-a labeled zone=c. We'll point a namespace-scoped ExternalDNS at it and watch a fresh record appear — without touching the existing a and b records:

kubectl apply -n app-a -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: c-gateway
  labels:
    app: approuting-demo
    zone: c
spec:
  gatewayClassName: approuting-istio
  listeners:
  - name: https
    hostname: c.${ZONE_NAME}
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        kubernetes.azure.com/tls-cert-keyvault-uri: $CERT_URI
        kubernetes.azure.com/tls-cert-service-account: $SA_NAME
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: c-route
spec:
  parentRefs:
  - name: c-gateway
  hostnames: ["c.${ZONE_NAME}"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF
kubectl wait -n app-a --for=condition=programmed gateway c-gateway --timeout=240s

Now create a namespace-scoped ExternalDNS in app-a with a label filter for zone=c:

kubectl apply -n app-a -f - <<EOF
apiVersion: approuting.kubernetes.azure.com/v1alpha1
kind: ExternalDNS
metadata:
  name: demo-ns-dns
spec:
  resourceName: demo-ns-dns
  dnsZoneResourceIDs:
  - $ZONE_ID
  resourceTypes:
  - gateway
  identity:
    type: workloadIdentity
    serviceAccount: $SA_NAME
  filters:
    gatewayLabels: "zone=c"
EOF

Two scoping rules apply at once:

1. Namespace scope — ExternalDNS only watches resources in its own namespace, so b-gateway in app-b is invisible to it.
2. Label filter — within app-a, only gateways carrying zone=c are picked up. The label key/value is rendered into the underlying external-dns controller's label selector.

After about a minute, a new c record appears alongside whatever was already there from the cluster-scoped run:

az network dns record-set a list --resource-group $RESOURCE_GROUP --zone-name $ZONE_NAME -o table

Name    ResourceGroup              Ttl    Type    AutoRegistered    Metadata
------  -------------------------  -----  ------  ----------------  --------
a       <your-rg>                  300    A       False
b       <your-rg>                  300    A       False
c       <your-rg>                  300    A       False

The new namespace-scoped controller is responsible for c only. a-gateway (also in app-a) is excluded by the label filter; b-gateway is excluded by the namespace filter.

5g. Verify HTTPS end-to-end​

Grab the zone's authoritative nameserver, resolve the FQDN through it, and feed the result straight into curl --resolve. That keeps the test self-contained — you don't need the registrar pointing at Azure DNS or your local resolver to know about the zone:

NS=$(az network dns zone show --resource-group $RESOURCE_GROUP --name $ZONE_NAME --query 'nameServers[0]' -o tsv | sed 's/\.$//')
GATEWAY_IP=$(dig +short @${NS} a.${ZONE_NAME} | tail -1)
echo "Resolved a.${ZONE_NAME} -> $GATEWAY_IP via $NS"

# Self-signed cert, so -k. Use --cacert if you exported the chain.
curl -k -I --resolve "a.${ZONE_NAME}:443:${GATEWAY_IP}" "https://a.${ZONE_NAME}/get"

You should see HTTP/2 200. The certificate Envoy presents is the one synced from Key Vault. You can re-tail the gateway deployment's logs (kubectl logs deploy/a-gateway-approuting-istio -n app-a) to see the access log line for the request.

Next steps​

• Migrate: If you're on the NGINX add-on, see the Ingress to Gateway API migration guide. Managed NGINX support continues through November 2026.
• Deepen: The full reference docs are at Configure ingress with the Kubernetes Gateway API and Secure ingress with the application routing Gateway API implementation.
• Track: AKS release tracker and the AKS release notes.
• Feedback: Open an issue at Azure/AKS issues.

Our vision for Azure Kubernetes Service has always been simple: AKS runs everywhere. The same Azure Kubernetes Service you run across Azure regions is the same platform you run in sovereign and private clouds, and on OEM hardware. One platform, every substrate — so that the skills, the tooling, and the operational model you already know come with you wherever your applications need to run.

Today, we're adding the newest member to that family — and it answers one of the strongest signals we've heard from customers at the edge. AKS now runs directly on bare-metal, small-form-factor devices at the edge, available today in public preview.

Why bare metal​

For years, customers have told us the same thing: the edge forces hard trade-offs on Kubernetes. They want the consistency of a managed Kubernetes platform in the places where their business actually happens — the retail back room, the factory floor, the quick-service restaurant, the branch office, the remote site. But the edge is not the datacenter. Space, power, and budget are constrained. Hardware is small and sometimes ruggedized. Connectivity is intermittent, and sites often need to keep running through disconnection. Given the Kubernetes control plane runs locally on the device, deployed workloads continue to operate normally during connectivity loss. Only portal visibility and Azure management actions are interrupted until the connection is restored.

What customers have asked us for is precise: run real Kubernetes directly on the hardware, with no hypervisor underneath and no virtualization layer to license or maintain; do it on lightweight edge hardware rather than datacenter-class servers; and let it keep operating through intermittent connectivity.

That is exactly what AKS on bare metal delivers. This is Azure Kubernetes Service running directly on the device with no hypervisor, on small-form-factor hardware, and resilient to intermittent connectivity. It is not a different product for the edge. It is the same AKS, supporting a new class of hardware at the edge.

From power-on to a running AKS cluster​

The experience is deliberately simple. On-site, someone plugs in a USB drive with an OS image from the Azure portal, powers on the device, waits a couple of minutes, and removes the drive. That's the only step that happens at the location. During this zero-touch provisioning process, a device voucher is written back to the USB drive and all configuration shifts to Azure — no deep infrastructure expertise required on site.

From there, everything happens in Azure. The machine is onboarded through the Azure Local small form factor creation experience — create a site, upload the voucher, and the machine comes up as Provisioned. The cluster creation looks exactly like AKS anywhere else: configure Azure RBAC, set the control-plane networking, enable Azure Monitor, and deploy. When it completes, you have an AKS cluster running on Azure Linux, on bare metal, fully manageable from Azure: workloads, services, ingress, and Kubernetes upgrades, all without leaving the portal.

One consistent AKS experience, from cloud to edge​

Provisioning is only half the story. What matters at the edge is how you operate the cluster over time — and here the principle is simple: AKS on bare metal is just an AKS cluster. There's no edge-specific console, no separate operational model, no exceptions to learn.

Because it's a first-class AKS cluster, it shows up in Azure Kubernetes Fleet Manager right alongside your cloud clusters, managed and monitored with the same controls and the same tooling you already use everywhere else. The same AKS, the same experience, consistent across cloud and edge — now including bare metal.

Available today​

This is what "AKS everywhere" was always meant to become: not a fragmented collection of edge-specific products and tools, but one Kubernetes platform that follows your applications to every place they need to run — now including the smallest, most constrained, and most remote hardware at the edge.

AKS on bare metal is now in public preview. We can't wait to see what you build on it.

Join the preview today at aka.ms/aks-edge-baremetal.

In the first three blog posts of this series, we introduced NVIDIA Dynamo on AKS, covered SLO-driven scaling with the Dynamo Planner and Profiler, and explored KV-cache-aware routing to establish the foundations for high-performance LLM serving. In this post, we move up one layer of the inference stack: how to describe and operate the distributed inference workload on a Kubernetes cluster.

As inference deployments evolve from simple model-serving replicas to systems with disaggregated prefill and decode phases, multi-node model instances, and explicit startup dependencies, an inference deployment benefits from a single API that describes the full serving pipeline: which roles exist, how they relate, and what constitutes a deployable unit of inference-serving capacity.

NVIDIA Grove addresses this layer of the stack with a Kubernetes-native API for describing an AI inference service — so the platform can schedule, start, and scale the deployment in units that match the inference architecture.

The Kubernetes orchestration gap for distributed inference​

Modern AI inference workloads are fundamentally different from the traditional distributed workloads built with independent, stateless microservices. A production LLM deployment may include distinct roles – frontend services, KV-aware routers, prefill and decode workers – that are tightly coupled in how they must be scheduled, started, and scaled. They require scheduling in groups, starting up in a specific order, and scaling in ratios that preserve an end-to-end serving path. For larger models, a single prefill or decode instance may span several pods; where a complete deployment of each instance is necessary for a functional serving capacity.

Kubernetes primitives can independently schedule and run these components, but they do not natively understand the inter-component relationships required for the deployment to operate as an inference service:

• Which pods form one usable model instance?
• Which roles must start before others?
• Which components scale independently, and which must scale as a group?

Using existing Kubernetes primitives alone, operations teams are forced to spread these considerations across bespoke manifests, labels, scripts, or custom controllers. This can create orchestration challenges and worsen the deployment observability.

LeaderWorkerSet (LWS) is an Upstream API that addresses a part of this problem by managing a leader pod and its worker pods as a replicated unit, which generally maps well to multi-node model instances. A full Dynamo-style deployment, however, can contain multiple leader-worker units across prefill and decode, plus routing and other service components. LWS may model those leader-worker groups, but it is not intended to describe the full inference service across them. Additionally, LWS couples multi-node scaling units with a leader-worker pattern, and not all multi-node use cases map to this cleanly. Overall, scheduling, startup, and scaling for such modern deployments require a more flexible Kubernetes-native workload abstraction.

NVIDIA Grove for Kubernetes-native AI scheduling​

NVIDIA Grove is an open-source Kubernetes API, available as a modular component of NVIDIA Dynamo, for defining both simple single-node and complex multi-node AI workloads. Where standard Kubernetes resources describe individual pods and services, Grove enables you to describe an entire inference serving system’s components and orchestration requirements as one object: routing, prefill, decode, leader-worker groups/multi-node units, startup dependencies, and scaling boundaries — all expressed in a single workload specification.

From that specification, Grove takes responsibility for coordinating the Kubernetes resources required to run the service. Scheduling, startup, and scaling have a shared view of the workload: which pods form a usable instance and which roles belong together. Scaling actions can target a single role, a complete multi-node model instance, or a full service replica. Startup dependencies between roles are part of the workload definition rather than external scripts.

The result is a workload API that matches how distributed inference systems are actually built: individual roles with different resource profiles, grouped model instances that must scale together, and complete service replicas that represent usable end-to-end serving capacity.

In an inference deployment, Dynamo provides the serving stack: inference workers, routing, the Planner, KV cache management, and backend integrations such as vLLM, TensorRT-LLM, and SGLang. Grove defines the Kubernetes orchestration layer around this stack, a.k.a. the contract between the inference system and the cluster. Grove can also be deployed standalone or integrated with other high-performance inference frameworks.

How Grove models the workload​

Grove's user-facing model is built around three hierarchical primitives that together describe the full shape of an inference deployment, from individual roles up to the complete service:

Using this hierarchy, Grove generates the lower-level Kubernetes resources the cluster needs to schedule, start, and scale the workload. Users work at the inference deployment level, while Grove translates the intent into Grove managed pods and other resources, including the scheduler-facing PodGang resources that encode the scheduling constraints.

Coordinating scheduling, startup, and scalability​

Once Grove has the workload specification, it uses the same hierarchy to guide how the workload is scheduled, started, and scaled.

Scheduling​

For multi-node inference, serving capacity arrives as a collection of complete instances. A prefill or decode instance that spans several pods can serve traffic only when the full set can run together. Gang scheduling gives Kubernetes an all-or-nothing contract: either all pods in the instance are scheduled together, or none are. Grove creates the scheduler-facing PodGang resources automatically from the workload specification, so this guarantee is part of the workload model rather than a rule that teams have to enforce separately.

Gang scheduling ensures that each individual component of a deployment is allocated as a complete unit, for example, a full prefill instance or a full decode instance. However, this alone does not guarantee that the deployment can actually serve end-to-end requests: the scheduler might schedule several complete prefill instances without any matching decode capacity, or vice versa.

Grove addresses this limitation with hierarchical gang scheduling. Instead of enforcing completeness only at the instance level, Grove also enforces it at the service level. The workload specifies higher-level viability constraints, for example, that at least one complete prefill instance and one complete decode instance must be scheduled together before the deployment is considered ready. The scheduler therefore treats these interdependent components as a “higher-level gang”: it schedules the required combination only when enough resources are available. This prevents the system from allocating individually complete but functionally unusable configurations that cannot process requests end to end.

On an AKS cluster, Grove-managed inference pods can be scheduled by a gang-aware scheduler like the KAI Scheduler, an open-source Kubernetes-native scheduler built for AI workloads with native PodGang awareness. Grove defines the inference workload - the roles, groups, dependencies, and scaling boundaries - while KAI gives Kubernetes the scheduling behavior needed to place those pods as complete units. Together, they enable these capabilities to be expressed and enforced cleanly within the Kubernetes-native infrastructure.

Startup​

Scheduling is only the first step. Distributed serving systems often have startup dependencies; for example, workers need to be discoverable before leaders start, and routers or frontends may need to be ready before model workers register. Grove integrates startup order with the workload definition through cliqueStartupType and startsAfter, instead of leaving it to scripts or external conventions.

Scaling​

Grove has multiple scaling boundaries in a hierarchy model: a PodClique for one role, a PodCliqueScalingGroup for a complete leader-worker unit, and a PodCliqueSet for the full service. This preserves independent scaling even when scheduling is coordinated: prefill and decode can be scheduled as viable units while still scaling separately in response to various bottlenecks.

Different scaling boundaries also integrate well with the Dynamo Planner (covered in Part 2 of this blog series). While the Planner reasons about traffic shape, sequence lengths, and time-to-first-token (TTFT) or inter-token latency (ITL) targets to decide when capacity should change, Grove ensures this decision targets the appropriate Grove workload boundary: a PodClique for a single role, a PodCliqueScalingGroup for a complete leader-worker instance, or the PodCliqueSet for the end-to-end inference service. Ultimately, Planner-driven scale-out doesn't just add pods; it adds a complete, gang-scheduled inference instance that is ready to serve.

Modeling inference deployments with Grove​

Grove's PodCliqueSet API is flexible enough to cover a range of inference deployment shapes:

Aggregated serving: where a single PodClique is sufficient; each worker handles the full request lifecycle, and scaling means adding more copies of that worker pod.

Disaggregated serving: router, prefill, and decode become separate PodCliques, each scaling independently in response to different traffic pressures.

Multi-node: when a single model instance spans multiple pods, a PodCliqueScalingGroup binds the leader and worker PodCliques into one scaling unit.

The Grove GitHub repository includes reference manifests for these deployment patterns, giving teams concrete starting points for mapping their own serving architecture into PodCliques, PodCliqueScalingGroups, and PodCliqueSets.

See Grove in action below as it orchestrates a multi-node Llama 3.1 70B deployment on AKS. The demo serves Llama-3.1-70B-Instruct-FP8 on 16x NVIDIA H100 (2 nodes of Azure Standard_ND96isr_H100_v5) with tp=16. The walkthrough shows a DynamoGraphDeployment being translated into Grove resources, worker pods landing across H100 nodes, and scale-out adding a second gang-scheduled two-pod worker as a grouped unit.

Looking ahead​

By introducing this workload layer, Grove provides Kubernetes with a structured contract for orchestrating the gang scheduling and scaling requirements of AI workloads. This also provides the foundation for handling complex challenges that modern inference workloads are facing today.

One such challenge is topology-aware scheduling. As deployments span multiple nodes and accelerator domains, scheduling decisions become critical to performance. Without topology awareness, gang-scheduled instances may be placed on nodes that are physically or network-topologically distant from one another, introducing communication bottlenecks that undermine the efficiency that the inference stack is designed to deliver.

Rolling updates introduce another layer of complexity. In disaggregated inference, prefill and decode components usually have compatibility constraints across framework versions, requiring cross-version traffic to be blocked during upgrades. This can strand capacity across versions: for example, a rollout might leave eight old-version prefill instances but only two old-version decode instances, reducing the amount of end-to-end capacity that can actually serve requests. Updating these systems safely requires rollout behavior that preserves viable serving capacity across the pipeline, rather than updating each role as an independent pool of pods.

In Part 5, we’ll build on the Grove workload model introduced here and explore how this structured view of the serving pipeline enables more advanced orchestration, including topology-aware scheduling for distributed inference and safer rolling updates for disaggregated serving pipelines. We’ll also share how we’re collaborating with the NVIDIA Grove and Dynamo communities toward a shared goal: building a Kubernetes-native orchestration layer for the demands of modern inference.

The approaches covered in this post apply mitigations as follows:

1. Immediate mitigation: deploy a DaemonSet using Fleet Manager resource placement. Best for clusters where a node image upgrade can't yet be applied.
2. Permanent mitigation: upgrade node images using Fleet Manager update runs. Applies patches when new node image version is available in the cluster's region.

The original AKS advisory and detailed mitigation guide for per-cluster application can be found in GitHub Issue 5753.

Before you begin​

• Azure CLI version 2.86.0 or later installed (instructions).
• Fleet Manager Azure CLI extension installed and updated.

az extension add --name fleet
az extension update --name fleet

• A Fleet Manager with a hub cluster (instructions). You can add a hub cluster if you are already using Fleet without one.
• AKS clusters added as member clusters with update group and member labels set. Groups and labels are required for controlling the rollout via placement and update runs.

az fleet member create \
  --resource-group $GROUP \
  --fleet-name $FLEET \
  --name $MEMBER_CLUSTER_NAME \
  --update-group canary \
  --labels upgroup=canary

• kubectl configured for the hub cluster (instructions).

Immediate mitigation - DaemonSet rollout​

For immediate protection, apply a DaemonSet that disables the vulnerable kernel module. The DaemonSet is deployed into its own Namespace on each cluster, making it easy to discern intent and easier to distribute across clusters. Given how DaemonSets function, the protection is applied to the node, even though we don't use the kube-system Namespace as shown in the original AKS mitigation guide.

Fleet Manager's cluster-scoped resource placement enables:

• Multi-cluster selection for a resource via ClusterResourcePlacement.
• Controlled rollout of the resource via ClusterStagedUpdateRun.

Step 1: Create the mitigation Namespace and DaemonSet​

Create the 01-kernel-lpe-mitigate.yaml file with the following contents. The DaemonSet blocks vulnerable modules covered by the CVEs.

apiVersion: v1
kind: Namespace
metadata:
  name: kernel-lpe-cve-mitigate-ns
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kernel-lpe-mitigate
  namespace: kernel-lpe-cve-mitigate-ns
  labels:
    app: kernel-lpe-mitigate
    purpose: security-mitigation
spec:
  selector:
    matchLabels:
      app: kernel-lpe-mitigate
  template:
    metadata:
      labels:
        app: kernel-lpe-mitigate
    spec:
      hostPID: true
      priorityClassName: system-node-critical
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - operator: Exists
      containers:
        - name: mitigate
          image: mcr.microsoft.com/cbl-mariner/busybox:2.0
          command:
            - /bin/sh
            - -c
            - |
              echo "=== Kernel LPE Module Mitigation ==="
              echo "Covers: CVE-2026-31431 (algif_aead), DirtyFrag (esp4/esp6/rxrpc)"

              MODULES="algif_aead esp4 esp6 rxrpc"

              for mod in $MODULES; do
                if ! grep -qs "install ${mod} /bin/false" /host/etc/modprobe.d/*.conf 2>/dev/null; then
                  printf "install %s /bin/false\nblacklist %s\n" "$mod" "$mod" >> /host/etc/modprobe.d/disable-kernel-lpe.conf
                  echo "Blocked ${mod}"
                else
                  echo "${mod} already blocked"
                fi

                if chroot /host grep -q "^${mod} " /proc/modules 2>/dev/null; then
                  if chroot /host modprobe -r "$mod" 2>/dev/null; then
                    echo "Unloaded ${mod}"
                  else
                    echo "WARNING: Could not unload ${mod} (in use). Reboot node."
                  fi
                fi
              done

              echo "=== Mitigation complete. Sleeping ==="
              sleep infinity
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 32Mi
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-root
              mountPath: /host
      volumes:
        - name: host-root
          hostPath:
            path: /
            type: Directory