Virtual accelerators such as NVIDIA vGPU are commonly used for smaller workloads because they allow a single physical GPU to be securely partitioned across multiple tenants or apps. This is especially valuable for enterprise AI/ML development environments, fine-tuning, and audio/visual processing. vGPU enables predictable performance profiles while still exposing CUDA capabilities to containerized workloads.

On Azure, the NVadsA10_v5 virtual machine (VM) series is backed by the physical NVIDIA A10 GPU in the host and offers this resource model. Instead of assigning the entire GPU to a single VM, the vGPU technology is used to partition the GPU into multiple fixed-size slices at the hypervisor layer.

In this post, we’ll walk through enabling the NVIDIA DRA driver on a node pool backed by an NVadsA10_v5 series vGPU on Azure Kubernetes Service (AKS).

Prepare your AKS cluster​

Verify DRA is enabled​

Starting with your AKS cluster running Kubernetes version 1.34 or above, you can confirm whether DRA is enabled on your cluster by looking for deviceclasses and resourceslices.

Check deviceclasses via kubectl get deviceclasses or check resourceslices via kubectl get resourceslices.

At this point, the results for both commands should look similar to:

No resources found

If DRA isn't enabled on your cluster (for example, if it is running an earlier Kubernetes version than 1.34), you may instead see an error like:

error: the server doesn't have a resource type "deviceclasses"/"resourceslices"

Add a vGPU node pool and label your nodes​

Add a GPU node pool and specify an Azure virtual machine (VM) size which supports virtualized accelerator workloads, such as NVadsA10_v5 series. These VM sizes are specifically designed for virtualized GPU scenarios, where each node receives a slice of a physical NVIDIA A10 rather than the entire card.

az aks nodepool add \
  --resource-group <resource-group> \
  --cluster-name <aks-cluster-name> \
  --name gpunodepool1 \
  --node-count 2 \
  --node-vm-size Standard_NV6ads_A10_v5

Today, AKS GPU nodes already include accelerator=nvidia, so we'll use this selector to apply the required label:

kubectl get nodes -l accelerator=nvidia -o name | \
  xargs -I{} kubectl label --overwrite {} nvidia.com/gpu.present=true

You can expect a similar output to the following:

node/aks-gpunodepool1-12345678-vmss000000 labeled
node/aks-gpunodepool1-12345678-vmss000001 labeled

Install the NVIDIA DRA driver​

The recommended way to install the driver is via Helm. Ensure you have Helm updated to the correct version.

Add the Helm chart that contains the DRA driver.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

Now, install the NVIDIA DRA driver version 25.8.1:

helm --install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
  --version="25.8.1" \
  --create-namespace \
  --namespace nvidia-dra-driver-gpu \
  --set "resources.gpus.enabled=true" \
  --set "gpuResourcesEnabledOverride=true" \
  --set "controller.nodeSelector=null" \
  --set "controller.tolerations[0].key=CriticalAddonsOnly" \
  --set "controller.tolerations[0].operator=Exists" \
  --set "controller.affinity=null" \
  --set "featureGates.IMEXDaemonsWithDNSNames=false"

Confirm that the following pods are running:

kubectl get pods -n nvidia-dra-driver-gpu

You can expect a similar output to the following:

NAME                                                  READY   STATUS    RESTARTS
nvidia-dra-controller-xxxxx                          1/1     Running   0
nvidia-dra-kubelet-plugin-aks-gpunodepool1-xxxxx     1/1     Running   0

At this stage, the NVIDIA DRA driver scans the node, detects the single vGPU device exposed by the Azure VM, and publishes it to the Kubernetes control plane as a DRA-managed device. Even though the underlying hardware is a shared A10, the driver registers one allocatable device on each node because that is what the VM presents.

Why do these Helm settings matter?​

Let's walk through some of the DRA Helm chart settings set earlier for vGPU:

1. resources.gpus.enabled=true

   Standard DRA workloads request devices via the gpu.nvidia.com device class, so resources.gpus.enabled=true is needed for GPU-accelerated workloads to schedule on these A10 nodes.

2. gpuResourcesEnabledOverride=true

   The Helm chart includes a validation guard to prevent collisions between the NVIDIA DRA driver (gpu.nvidia.com) and legacy NVIDIA device plugin (nvidia.com/gpu extended resource). Since we are running DRA exclusively (with no legacy device plugin), we bypass the validation: gpuResourcesEnabledOverride=true to ensure the chart installs successfully.

3. featureGates.IMEXDaemonsWithDNSNames=false

   This feature gate is enabled by default and requires NVIDIA GRID GPU driver version >= 570.158.01. For Azure VM sizes, like the A10 series, that require the distinct GRID driver 550 branch today, we explicitly set featureGates.IMEXDaemonsWithDNSNames=false to disable IMEX for this GPU size.

Verify DeviceClass and ResourceSlices​

After deployment, confirm that the gpu.nvidia.com DeviceClass exists:

kubectl get deviceclasses

Expected output:

NAME             DRIVER
gpu.nvidia.com   nvidia.com/dra

Check ResourceSlices:

kubectl get resourceslices

Expected output:

NAME                                  NODE
gpu-aks-gpunodepool1-xxxxx-0          aks-gpunodepool1-xxxxx-vmss000000

Now, we’ve confirmed that the DRA driver discovered and published our vGPU-backed resources, and the nodes are ready to accept workloads! You can follow these steps to run a sample workload using the DRA specifications.

vGPU options in Azure​

Beyond Standard_NV6ads_A10_v5 (one-sixth of an A10 GPU), this VM series offers larger fractional profiles:

• Standard_NV12ads_A10_v5: one-third of a physical A10, with 8 GB of accelerator memory
• Standard_NV18ads_A10_v5: one-half of a physical A10, with 12 GB of accelerator memory

These sizes (and more) in the NVadsA10_v5 VM series map to a fixed NVIDIA vGPU profile and the fraction determines how much GPU memory and compute capacity (serial multiprocessors) the VM receives. The limits are enforced at the hypervisor layer, so AKS ultimately sees a single GPU device with predictable, guaranteed capacity.

Looking ahead​

As GPUs become first-class resources in Kubernetes, combining virtualized GPU with DRA provides a practical way to run shared, production-grade workloads on AKS. vGPU supported Azure VM series offer partial GPUs for scenarios such as media rendering and transcoding and fine-tuning small to medium language models, while DRA ensures those resources are allocated explicitly and scheduled with awareness of real cluster state.

For large AKS deployments, especially in regulated or cost-sensitive industries, getting GPU placement and utilization right directly affects job throughput and infrastructure efficiency. Using DRA with vGPU will enable organizations to move beyond coarse node-level allocation toward controlled, workload-driven GPU consumption at scale.

By default, Kubernetes schedules GPUs as entire units; when a workload requires only a fraction of a GPU, the remaining capacity can remain unused. Over time, this leads to lower hardware utilization and higher infrastructure costs within a cluster.

Multi-instance GPU (MIG) combined with dynamic resource allocation (DRA) helps address this challenge. MIG partitions a physical GPU into isolated instances with dedicated compute and memory resources, while DRA enables those instances to be provisioned and bound dynamically through Kubernetes resource claims. Rather than treating a GPU as an indivisible resource, the cluster can allocate right-sized GPU partitions to multiple workloads at the same time!

In this post, we walk through how to configure MIG with the NVIDIA GPU Operator on AKS, enable the NVIDIA DRA driver, define the necessary Kubernetes resource abstractions, and deploy a workload that consumes a MIG-backed GPU instance (NVIDIA GPUs - such as the A100, H100, H200 and more - that support partitioning can be found in the MIG User Guide).

Prepare your AKS cluster​

Starting with your AKS cluster running Kubernetes version 1.34 or above, you can confirm whether DRA is enabled on your cluster by looking for deviceclasses and resourceslices.

Check deviceclasses via kubectl get deviceclasses or check resourceslices via kubectl get resourceslices.

At this point, the results for both commands should look similar to:

No resources found

If DRA isn't enabled on your cluster (for example, if it is running an earlier Kubernetes version than 1.34), you may instead see an error like:

error: the server doesn't have a resource type "deviceclasses"/"resourceslices"

Set up NVIDIA GPU Operator​

We’ll leverage the NVIDIA GPU Operator to manage the GPU driver lifecycle. When creating the GPU-enabled node pool, specify --gpu-driver none to prevent preinstalled drivers from conflicting with the operator-managed stack and ensure consistent configuration across nodes. In this example, we provision an AKS node pool with an Azure NDm_A100_v4 VM size supporting MIG:

az aks nodepool add \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name gpunp \
    --node-count 1 \
    --gpu-driver none \
    --node-vm-size Standard_ND96amsr_A100_v4

Next, install the NVIDIA GPU Operator with MIG enabled and the legacy Kubernetes device plugin disabled. We consolidate these configuration settings in a YAML file named operator-install.yaml as follows:

mig:
  strategy: single
devicePlugin:
  enabled: false
driver:
  enabled: true
toolkit:
  env:
    # Limits containers running in unprivileged mode from requesting access to arbitrary GPU devices 
    - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
      value: "false"

The single strategy partitions each GPU into uniform partitions; alternatively, you can configure mixed strategy to partition each GPU into distinct resource types. After preparing and saving the configuration file, install the operator with Helm:

helm install --wait \
  --generate-name -n gpu-operator \
  --create-namespace \
  nvidia/gpu-operator \
  --version=v25.10.0 \
  -f operator-install.yaml

Now, the operator has installed the GPU driver, configured single-strategy MIG, and prepared the node pool for GPU partitioning.

Install the NVIDIA DRA driver​

DRA introduces a more flexible device management model in Kubernetes; instead of statically advertising a fixed number of GPUs, DRA allows workloads to create and bind resource claims dynamically.

The NVIDIA DRA driver installation enables GPU resources and points to the driver root managed by the operator, as shown in the following configuration file named dra-install.yaml:

gpuResourcesEnabledOverride: true
resources-computeDomains:
  enabled: false # We'll be using GPUs, not compute domains.
controller:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.azure.com/mode
            operator: In
            values:
            - system   # Makes sure the system nodes are utilized 
nvidiaDriverRoot: "/run/nvidia/driver"

Using the above settings, install the NVIDIA DRA driver in a dedicated namespace:

helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
--version="25.8.1" \
--create-namespace \
--namespace nvidia-dra-driver-gpu \
-f dra-install.yaml

Verify MIG configuration​

Before scheduling workloads, confirm that the AKS node recognizes the GPU and that MIG is active. Inspecting the node should show that a GPU is present and that MIG capability and strategy are correctly applied. You should see indicators such as nvidia.com/mig.capable=true and nvidia.com/mig.strategy=single, along with a successful MIG configuration state.

kubectl describe node aks-gpunp-12340814-vmss000000 | grep "gpu"

Output:

Name:               aks-gpunp-12340814-vmss000000
                    agentpool=gpunp
                    kubernetes.azure.com/agentpool=gpunp
                    kubernetes.io/hostname=aks-gpunp-12340814-vmss000000
                    nvidia.com/gpu-driver-upgrade-state=upgrade-done
                    nvidia.com/gpu.compute.major=9
                    nvidia.com/gpu.compute.minor=0
                    nvidia.com/gpu.count=1 # GPUs are recognized

kubectl describe node aks-gpunp-12340814-vmss000000 | grep "mig"

Example result:

Name:               aks-gpunp-12340814-vmss000000
...
...
                    nvidia.com/gpu.deploy.mig-manager=true
                    nvidia.com/mig.capable=true
                    nvidia.com/mig.config=all-disabled
                    nvidia.com/mig.config.state=success
                    nvidia.com/mig.strategy=single

Your AKS cluster should now be ready to expose MIG-enabled GPU partitions as dynamically allocatable devices!

Define a DeviceClass​

DRA introduces a DeviceClass abstraction that allows Kubernetes to select devices based on accelerator type and characteristics. In this case, we define a class that selects NVIDIA GPUs:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: nvidia-mig
spec:
  selectors:
  - cel:
      expression: "device.driver == 'gpu.nvidia.com'"

This definition tells AKS that any request referencing nvidia-mig should resolve to devices managed by the NVIDIA GPU driver.

kubectl get deviceclass

NAME                                        AGE
compute-domain-daemon.nvidia.com            ...
compute-domain-default-channel.nvidia.com   ...
gpu.nvidia.com                              ...
mig.nvidia.com                              ... 
nvidia-mig                                  1m31s

Create a MIG ResourceClaimTemplate​

Instead of requesting nvidia.com/gpu: 1 in a pod spec, workloads will now reference a ResourceClaimTemplate, which describes the device requirement declaratively. We'll apply a MIG ResourceClaimTemplate to the AKS cluster as follows:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-gpu-1g
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: nvidia-mig
          count: 1

This abstraction decouples workloads from physical device details. A job does not need to know which GPU or partition it receives—it just declares its need for a device from the nvidia-mig class.

Deploy a sample MIG workload​

To validate the setup, we can deploy a GPU-accelerated workload requesting a MIG partition. Our example below uses a TensorFlow sample and generally mirrors how a data processing or video transcoding job can consume a resource partition in production environments:

apiVersion: batch/v1
kind: Job
metadata:
  name: samples-tf-mnist-demo
  labels:
    app: samples-tf-mnist-demo
spec:
  template:
    metadata:
      labels:
        app: samples-tf-mnist-demo
    spec:
      restartPolicy: OnFailure
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      containers:
      - name: samples-tf-mnist-demo
        image: mcr.microsoft.com/azuredocs/samples-tf-mnist-demo:gpu
        imagePullPolicy: IfNotPresent
        args: ["--max_steps", "500"]
        resources:
          claims:
          - name: gpu
      resourceClaims:
      - name: gpu
        resourceClaimTemplateName: mig-gpu-1g

After deploying this job, we can check its status and the usage of the mig-gpu-1g resource claim template we previously created:

kubectl get job

NAME                    STATUS    COMPLETIONS   DURATION   AGE
samples-tf-mnist-demo   Running   0/1           2m59s      2m59s

kubectl get resourceclaimtemplate

NAME         AGE
mig-gpu-1g   11s

The key difference from traditional GPU scheduling is the use of resources.claims and resourceClaimTemplateName: Kubernetes coordinates with the DRA driver to provision and bind a MIG partition dynamically. Now when multiple jobs are submitted, each can receive its own isolated instance, allowing parallel execution on the same physical GPU.

A more elastic GPU future on AKS​

GPUs in Kubernetes have traditionally been scheduled as indivisible units. By enabling MIG and DRA on AKS, you move toward a model where accelerators are elastic, shareable, and first-class resources in the control plane. For organizations running parallel workloads that only partially utilize GPU capacity, this shift unlocks immediate cost efficiency and operational benefits.

If you are already operating GPU-enabled node pools on AKS and notice underutilization, implementing MIG with Dynamic Resource Allocation is a highly impactful architectural improvement you can make. It allows you to run more workloads on the same hardware while maintaining predictability and cloud-native operational simplicity.

Additional resources​

To learn more about NVIDIA MIG and DRA, check out the following resources:

• MIG User Guide
• Additional MIG profiles per NVIDIA GPU series
• Types of DRA users on Kubernetes

Ray is an open-source distributed compute framework for scaling Python and AI workloads from a laptop to clusters with thousands of nodes. Anyscale provides a managed ML/AI platform and an optimized Ray runtime with better scalability, observability, and operability than running open-source KubeRay—including intelligent autoscaling, enhanced monitoring, and fault-tolerant training.

As part of Microsoft and Anyscale's strategic collaboration to deliver distributed AI/ML Azure-native computing at scale, we've been working closely with Anyscale to enhance the production-readiness of Ray workloads on Azure Kubernetes Service (AKS) in three critical areas:

• Elastic scalability through multi-cluster multi-region capacity aggregation
• Data persistence with unified storage across ML/AI development and operation lifecycle
• Operational simplicity through automated credential management with service principal

Whether you're fine-tuning models with DeepSpeed or LLaMA-Factory or deploying inference endpoints for LLMs ranging from small to large-scale reasoning models, Anyscale on AKS delivers a production-grade ML/AI platform that scales with your needs.

Multi-cluster Multi-region​

GPU scarcity remains one of the most significant challenges in large-scale ML operations. High-demand accelerators like NVIDIA GPUs often face capacity constraints in specific Azure regions, leading to delays in provisioning clusters or launching training jobs.

By deploying Ray clusters across multiple AKS clusters in different Azure regions, you can:

• Increase GPU availability: Distribute workloads across clusters and regions with available capacity, reducing wait times for cluster provisioning
• Scale beyond single-cluster limits: Azure imposes quota limits on GPU instances per region, but multi-region deployments let you aggregate capacity
• Improve fault tolerance: If one region experiences an outage or capacity shortage, workloads can be automatically rerouted to healthy clusters

With infrastructure deployed across multiple regions—and optionally on-premises or other cloud environments—you can manage and monitor Anyscale workloads on registered clusters from the Anyscale console. This multi-cloud and hybrid cloud approach lets you access GPU capacity wherever it exists, whether in Azure regions, on-premises data centers, or other cloud providers. Extend your compute pool beyond Azure by connecting on-premises GPU clusters through AKS enabled by Azure Arc, aggregating existing infrastructure investments with cloud-based resources:

Anyscale Workspaces provides a managed environment for running interactive Ray workloads, with manual or automatic scheduling across available clusters based on resource requirements:

To add a cluster or another region to your existing Anyscale cloud, define a cloud resource as below cloud_resource.yaml:

name: k8s-azure-$REGION
provider: AZURE
compute_stack: K8S
region: $REGION
object_storage:
  bucket_name: abfss://${STORAGE_CONTAINER}@${STORAGE_ACCOUNT}.dfs.core.windows.net
file_storage:
  persistent_volume_claim: blob-fuse2
azure_config:
  tenant_id: ${AZURE_TENANT_ID}
kubernetes_config:
  anyscale_operator_iam_identity: ${IDENTITY_PRINCIPAL_ID}

Then create the cloud resource using the Anyscale CLI:

anyscale cloud resource create \
  --cloud "$ANYSCALE_CLOUD_NAME" \
  -f cloud_resource.yaml

Unified Storage​

Another major challenge is sharing training data, model checkpoints, and artifacts across the ML/AI workflow—from pre-training to fine-tuning to inference. Azure BlobFuse2 mounts Azure Blob Storage into Ray worker pods as a shared POSIX filesystem, providing unified, cloud-scale storage beyond workload lifecycle.

From Ray's perspective, BlobFuse2 is just a mounted filesystem. Ray tasks and actors read datasets and write checkpoints via normal file I/O, while BlobFuse2 ensures data is persisted to Azure Blob Storage and shared across pods. This keeps Ray code portable while benefiting from Azure-native storage. By decoupling data from compute, you can scale Ray clusters up and down across node pools without data loss, while local caching prevents GPU stalls during large training job run.

To use BlobFuse2 with Ray on AKS:

1. Create Azure Blob Storage containers for datasets, checkpoints, and models.

2. Enable blob-csi-driver when creating your AKS cluster:

   az aks create \
     ...
     --enable-blob-driver
     ...
   

3. Create a StorageClass that uses workload identity authentication and optimized caching parameters for large files:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: blob-fuse2
   
   provisioner: blob.csi.azure.com
   parameters:
     protocol: fuse2
     storageAccount: ${STORAGE_ACCOUNT}
     resourceGroup: ${RESOURCE_GROUP}
     clientID: ${IDENTITY_CLIENT_ID}
     mountWithWorkloadIdentityToken: "true"
   
   mountOptions:
     - -o allow_other
     - --file-cache-timeout-in-seconds=120
     - --use-attr-cache=true
     - --cancel-list-on-mount-seconds=10
     - -o attr_timeout=120
     - -o entry_timeout=120
     - -o negative_timeout=120
     - --log-level=LOG_WARNING
     - --cache-size-mb=1000
   
   allowVolumeExpansion: true
   reclaimPolicy: Retain
   volumeBindingMode: Immediate
   

4. Create a PersistentVolumeClaim in the anyscale-operator namespace with ReadWriteMany access mode. This allows multiple Ray workers across different nodes to access the same data:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: blob-fuse2
     namespace: anyscale-operator
   spec:
     accessModes:
       - ReadWriteMany
     storageClassName: blob-fuse2
     resources:
       requests:
         storage: 100Gi
   

5. Configure Ray workloads to read from and write to mounted path /mnt/cluster_storage.

With this setup, Ray workers read and write data using standard POSIX file operations while benefiting from the durability and scalability of Azure Blob Storage. ML/AI engineers can seamlessly transition from pre-training to fine-tuning to inference without manual data migration.

Service Principal Authentication​

Maintaining secure and reliable authentication between Ray clusters and Azure resources can be challenging. Previous integration relied on CLI tokens or API keys that expire every 30 days, requiring manual rotation or causing potential service disruptions.

By combining Microsoft Entra service principals with managed identities (workload identity), you eliminate this operational burden without storing long-lived secrets in the cluster. Pods use the managed identity to obtain short-lived access tokens for the service principal from Microsoft Entra ID, and Entra automatically refreshes these tokens as needed:

• No long-lived credentials (client secrets or certificates) are stored in Kubernetes clusters
• Automatic short-lived token issuance and refresh without manual intervention
• Fine-grained RBAC for Azure resource access
• Full audit trails through Azure Activity Logs

The following diagram illustrates how the service principal enables the Anyscale Kubernetes Operator to authenticate without storing credentials:

In this authentication flow:

1. The Anyscale Operator pod authenticates using a user-assigned managed identity.
2. The managed identity requests an access token with scope api://086bc.../.default.
3. The token is issued by the Anyscale Kubernetes Operator Auth service principal.
4. The service principal's appId becomes the AZURE_CLIENT_ID environment variable.
5. The managed identity's appId appears as the oid claim in the resulting access token.

This approach removes the need for manual credential rotation by letting Azure automatically manage token lifecycle, reducing operational overhead and minimizing the risk of authentication failures due to expired secrets. In a multi-cluster environment, this becomes even more critical—automated credential management across clusters simplifies operations and ensures consistent, secure authentication at scale.

Conclusion​

Running Ray at scale on Azure Kubernetes Service requires careful consideration of compute, storage, and security strategies. Whether you're running hundreds of small experiments or large, multi-day training jobs, Anyscale on AKS gives you the elastic scale, unified storage, and operational simplicity to take ML/AI workloads from development to production.

To explore private preview access for Anyscale on AKS, contact your Microsoft account team or open a request on the AKS GitHub repository with details about your Ray workloads and target regions.

KubeVirt is a Cloud Native Computing Foundation (CNCF) incubating open-source project that allows users to run, deploy, and manage VMs in their Kubernetes clusters.

In this post, you will learn how KubeVirt lets you run, deploy, and manage VMs in AKS.

Why KubeVirt matters​

KubeVirt can help organizations that are in various stages of their Kubernetes journey manage their infrastructure more effectively. It allows customers to manage legacy VM workloads alongside containerized applications using the same Kubernetes API.

VMs deployed on KubeVirt act much the same way as VMs deployed in more traditional manners would but can run and be managed alongside other containerized applications through traditional Kubernetes tools. Capabilities like scheduling that users are familiar with on Kubernetes can also be applied to these VMs.

Management of these otherwise disparate deployments can be simplified and unified. This unified management can help teams avoid the sprawl that would otherwise come with managing multiple platforms.

The capability to mix and match your workloads in a "hybrid" setting can also allow organizations that might have more complex, legacy VM-based applications to incrementally transition to containers while ensuring their applications remain operational throughout the transition.

Deploying KubeVirt on AKS​

You can deploy KubeVirt on any AKS cluster that has nodes running VM SKUs that support nested virtualization.

Prerequisites​

• KubeVirt on AKS requires a chosen VM SKU to support nested virtualization. You can confirm support on the VM size's Microsoft Learn page, such as Standard_D4s_v5. Using the Standard_D4s_v5 SKU as an example, on the SKU page, you can see whether or not nested virtualization is supported in the "Feature support" section.
• Install the virtctl binary utility to better access and control your VirtualMachineInstances. You can follow instructions on the KubeVirt page to install virtctl.

Creating an AKS cluster​

1. Create your AKS cluster.

   az aks create --resource-group <resource-group> --name <cluster-name> --node-vm-size Standard_D4s_v5
   

2. After your cluster is up and running, get the access credentials for the cluster.

   az aks get-credentials --resource-group <resource-group> --name <cluster-name>
   

Installing KubeVirt​

1. Install the KubeVirt operator.

   # Get the latest release
   export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
   
   # Deploy the KubeVirt operator
   curl -L https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml | \
   sed '8249,8254c\            nodeSelectorTerms:\n            - matchExpressions:\n              - key: node-role.kubernetes.io/worker\n                operator: DoesNotExist' | \
   kubectl apply -f -
   

2. Install the KubeVirt custom resource.

   curl -L https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml \
   | yq '.spec.infra.nodePlacement={}' \
   | kubectl apply -f -
   

   Notice the empty nodePlacement: {} and the update for the node selector. By default, KubeVirt sets the node-affinity of operator/custom resource components to control plane nodes. Because AKS control plane nodes are fully managed by Azure and inaccessible to KubeVirt, this update to utilize worker nodes avoids potential failures.

Verify KubeVirt installation​

Once all the components are installed, you can confirm that all KubeVirt components are up and running properly in your cluster:

kubectl get pods -n kubevirt -o wide

You should see something like this:

NAME                               READY   STATUS    RESTARTS   AGE     IP             NODE                                NOMINATED NODE   READINESS GATES
virt-api-7f7d56bbc5-s9nr4          1/1     Running   0          4m10s   10.244.0.174   aks-nodepool1-26901818-vmss000000   <none>           <none>
virt-controller-7c5744f574-56dd5   1/1     Running   0          3m39s   10.244.0.204   aks-nodepool1-26901818-vmss000000   <none>           <none>
virt-controller-7c5744f574-ftz6z   1/1     Running   0          3m39s   10.244.0.120   aks-nodepool1-26901818-vmss000000   <none>           <none>
virt-handler-dlkxf                 1/1     Running   0          3m39s   10.244.0.52    aks-nodepool1-26901818-vmss000000   <none>           <none>
virt-operator-7c8bdfb574-54cs6     1/1     Running   0          9m38s   10.244.0.87    aks-nodepool1-26901818-vmss000000   <none>           <none>
virt-operator-7c8bdfb574-wzdxt     1/1     Running   0          9m38s   10.244.0.153   aks-nodepool1-26901818-vmss000000   <none>           <none>

Creating VirtualMachineInstance resources in KubeVirt​

With KubeVirt installed on your cluster, you can now create your VirtualMachineInstance (VMI) resources.

1. Create your VMI. Save the following YAML, which will create a VMI based on Fedora OS, as vmi-fedora.yaml. The username for this deployment will default to fedora, while you can specify a password of your choosing in password: <my_password>.

   apiVersion: kubevirt.io/v1
   kind: VirtualMachineInstance
   metadata:
     labels:
       special: vmi-fedora
     name: vmi-fedora
   spec:
     domain:
       devices:
         disks:
         - disk:
             bus: virtio
           name: containerdisk
         - disk:
             bus: virtio
           name: cloudinitdisk
         interfaces:
         - masquerade: {}
           name: default
         rng: {}
       memory:
         guest: 1024M
       resources: {}
     networks:
     - name: default
       pod: {}
     terminationGracePeriodSeconds: 0
     volumes:
     - containerDisk:
         image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel
       name: containerdisk
     - cloudInitNoCloud:
         userData: |-
           #cloud-config
           password: <my_password>
           chpasswd: { expire: False }
       name: cloudinitdisk
   

2. Deploy the VMI in your cluster.

   kubectl apply -f vmi-fedora.yaml
   

   If successful, you should see an output similar to virtualmachineinstance.kubevirt.io/vmi-fedora created.

Check out the created VMI​

1. Test and make sure the VMI is created and running via kubectl get vmi. You should see a result similar to:

   NAME         AGE   PHASE     IP             NODENAME                            READY
   vmi-fedora   85s   Running   10.244.0.213   aks-nodepool1-26901818-vmss000000   True
   

2. Connect to the newly created VMI and inspect it.

   virtctl console vmi-fedora
   

   When prompted with credentials, the default username is fedora, while the password was configured in vmi-fedora.yaml.

   vmi-fedora login: fedora
   Password: 
   

   Once logged in, run cat /etc/os-release to display the OS details.

   [fedora@vmi-fedora ~]$ cat /etc/os-release
   NAME=Fedora
   VERSION="32 (Cloud Edition)"
   ID=fedora
   VERSION_ID=32
   VERSION_CODENAME=""
   PLATFORM_ID="platform:f32"
   PRETTY_NAME="Fedora 32 (Cloud Edition)"
   ANSI_COLOR="0;34"
   LOGO=fedora-logo-icon
   CPE_NAME="cpe:/o:fedoraproject:fedora:32"
   HOME_URL="https://fedoraproject.org/"
   DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
   SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
   BUG_REPORT_URL="https://bugzilla.redhat.com/"
   REDHAT_BUGZILLA_PRODUCT="Fedora"
   REDHAT_BUGZILLA_PRODUCT_VERSION=32
   REDHAT_SUPPORT_PRODUCT="Fedora"
   REDHAT_SUPPORT_PRODUCT_VERSION=32
   PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
   VARIANT="Cloud Edition"
   VARIANT_ID=cloud
   

Converting your VMs​

At this point, you should have KubeVirt up and running in your AKS cluster and a VMI deployed. KubeVirt can help with a plethora of scenarios that operational teams may run into. Migrating legacy VMs to KubeVirt can be an involved process, however. Doing it manually involves steps like converting the VM disk and persisting a VM disk to creating a VM template.

Tools like Forklift can automate some of the complexity involved with the migration. Forklift allows VMs to be migrated at scale to KubeVirt. The migration can be done by installing Forklift custom resources and setting up their respective configs in the target cluster. Some great walkthroughs of VM migration can be found in these videos detailing how Forklift helps deliver a better UX when importing VMs to KubeVirt and breaking down everything from the architecture to a demo of Forklift 2.0.

Running in production​

When running production grade workloads, stability of both the KubeVirt components and the individual VMs can also be a point of consideration. As we hinted at earlier, KubeVirt typically sets the node-affinity of operator/custom resource components to control-plane nodes. In our deployment, we have the KubeVirt components running on worker nodes.

In order to maintain a control-plane/worker node split, it can be advisable to aim to deploy KubeVirt components in an agentpool that can be designated as the "control-plane" node, while VMs spun up can be ran in designated "worker node" agentpools.

KubeVirt is currently not an officially supported AKS addon/extension, so there is no Microsoft backed SLA/SLO in place for KubeVirt deployments in AKS. If customers need an officially supported offering, Azure Red Hat OpenShift is a generally available platform to manage virtualized and containerized applications together.

Share your feedback​

If you're using KubeVirt on AKS or are interested in trying it, we'd love to hear from you! Your feedback will help the AKS team plan how to best support these types of workloads on our platform. Share your thoughts in our GitHub Issue.

Resources​

• What is KubeVirt?
• KubeVirt user guides
• Roadmap item for KubeVirt on AKS

Introduction​

LLM inference service is a basic and widely used feature in KAITO. As the number of waiting inference requests increases, scale more inference instances to prevent blocking. Conversely, reduce inference instances when requests decline to improve GPU resource utilization. Kubernetes Event Driven Autoscaling (KEDA) is well-suited for inference pod autoscaling. It enables event-driven, fine-grained scaling based on external metrics and triggers. KEDA supports a wide range of event sources (like custom metrics), allowing pods to scale precisely in response to workload demand. This flexibility and extensibility make KEDA ideal for dynamic, cloud-native applications that require responsive and efficient autoscaling.

To enable intelligent autoscaling for KAITO inference workloads using service monitoring metrics, utilize the following components and features:

• Kubernetes Event Driven Autoscaling (KEDA)

• KEDA KAITO Scaler: A dedicated KEDA external scaler, eliminating the need for external dependencies such as Prometheus.

• KAITO InferenceSet CustomResourceDefinition (CRD) and controller: A new CRD and controller were built on top of the KAITO workspace for intelligent autoscaling, introduced as an alpha feature in KAITO version v0.8.0.

Architecture​

The following diagram shows how KEDA KAITO Scaler integrates KAITO InferenceSet with KEDA to autoscale inference workloads on AKS:

Getting started​

Create an AKS cluster with GPU auto-provisioning capabilities for KAITO​

Refer to the instructions on how to create an AKS cluster with GPU auto-provisioning capabilities for KAITO.

Enable InferenceSet controller in KAITO​

The InferenceSet CRD and controller were introduced as an alpha feature in KAITO version v0.8.0. Built on top of the KAITO workspace, InferenceSet supports the scale subresource API for intelligent autoscaling. To use InferenceSet, the InferenceSet controller must be enabled during the KAITO installation.

export CLUSTER_NAME=kaito

helm repo add kaito https://kaito-project.github.io/kaito/charts/kaito
helm repo update
helm upgrade --install kaito-workspace kaito/workspace \
  --namespace kaito-workspace \
  --create-namespace \
  --set clusterName="$CLUSTER_NAME" \
  --set featureGates.enableInferenceSetController=true \
  --wait

Install KEDA​

• Option 1: Enable managed KEDA add-on For instructions, refer to Install KEDA add-on on AKS

• Option 2: Install KEDA using Helm chart

The following example demonstrates how to install KEDA 2.x using Helm chart. For instructions on installing KEDA through other methods, refer to the KEDA deployment documentation.

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace kube-system

Example Scenarios​

Time-Based KEDA Scaler​

The KEDA cron scaler enables scaling of workloads according to time-based schedules, making it especially beneficial for workloads with predictable traffic patterns. It is perfect for situations where peak hours are known ahead of time, allowing you to proactively adjust resources before demand rises. For more details about time-based scalers, refer to Scale applications based on a cron schedule.

Example: Business Hours Scaling​

• Create a KAITO InferenceSet for running inference workloads

The following example creates an InferenceSet for the phi-4-mini model:

cat <<EOF | kubectl apply -f -
apiVersion: kaito.sh/v1alpha1
kind: InferenceSet
metadata:
  name: phi-4-mini
  namespace: default
spec:
  labelSelector:
    matchLabels:
      apps: phi-4-mini
  replicas: 1
  template:
    inference:
      preset:
        accessMode: public
        name: phi-4-mini-instruct
    resource:
      instanceType: Standard_NC24ads_A100_v4
EOF

• Create a KEDA ScaledObject

Below is an example of creating a ScaledObject that scales a KAITO InferenceSet based on business hours:

• Scale up to 5 replicas from 6:00 AM to 8:00 PM (peak hours)

• Scale down to 1 replica otherwise (off-peak hours)

cat <<EOF | kubectl apply -f -
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kaito-business-hours-scaler
  namespace: default
spec:
  # Target KAITO InferenceSet to scale
  scaleTargetRef:
    apiVersion: kaito.sh/v1alpha1
    kind: InferenceSet
    name: phi-4-mini
  # Scaling boundaries
  minReplicaCount: 1
  maxReplicaCount: 5
  # Cron-based triggers for time-based scaling
  triggers:
  # Scale up to 5 replicas at 6:00 AM (start of business hours)
  - type: cron
    metadata:
      timezone: "America/New_York"  # Adjust timezone as needed
      start: "0 6 * * 1-5"          # 6:00 AM Monday to Friday
      end: "0 20 * * 1-5"           # 8:00 PM Monday to Friday
      desiredReplicas: "5"          # Scale to 5 replicas during business hours
  # Scale down to 1 replica at 8:00 PM (end of business hours)
  - type: cron
    metadata:
      timezone: "America/New_York"  # Adjust timezone as needed
      start: "0 20 * * 1-5"         # 8:00 PM Monday to Friday
      end: "0 6 * * 1-5"            # 6:00 AM Monday to Friday (next day)
      desiredReplicas: "1"          # Scale to 1 replica during off-hours
EOF

Metric-Based KEDA Scaler​

Install KEDA KAITO Scaler​

This component is required only when using metric-based KEDA scaler, ensure that KEDA KAITO Scaler is installed within the same namespace as KEDA.

helm repo add keda-kaito-scaler https://kaito-project.github.io/keda-kaito-scaler/charts/kaito-project
helm upgrade --install keda-kaito-scaler -n kube-system keda-kaito-scaler/keda-kaito-scaler

After a few seconds, the keda-kaito-scaler deployment starts.

# kubectl get deployment keda-kaito-scaler -n kube-system
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
keda-kaito-scaler   1/1     1            1           28h

The keda-kaito-scaler provides a simplified configuration interface for scaling vLLM inference workloads, it directly scrapes metrics from inference pods, eliminating the need for a separate monitoring stack.

Example: Create a KAITO InferenceSet with annotations for running inference workloads​

• The following example creates an InferenceSet for the phi-4-mini model, using annotations with the prefix scaledobject.kaito.sh/ to supply parameter inputs for the KEDA KAITO scaler.

  • scaledobject.kaito.sh/auto-provision
    • required, when set to true, the KEDA KAITO scaler automatically provisions a ScaledObject based on the InferenceSet object
  • scaledobject.kaito.sh/max-replicas
    • required, maximum number of replicas for the target InferenceSet
  • scaledobject.kaito.sh/metricName
    • optional, specifies the metric name collected from the vLLM pod, which is used for monitoring and triggering the scaling operation, default is vllm:num_requests_waiting, find all vllm metrics in vLLM Production Metrics
  • scaledobject.kaito.sh/threshold
    • required, specifies the threshold for the monitored metric that triggers the scaling operation

cat <<EOF | kubectl apply -f -
apiVersion: kaito.sh/v1alpha1
kind: InferenceSet
metadata:
  annotations:
    scaledobject.kaito.sh/auto-provision: "true"
    scaledobject.kaito.sh/max-replicas: "5"
    scaledobject.kaito.sh/metricName: "vllm:num_requests_waiting"
    scaledobject.kaito.sh/threshold: "10"
  name: phi-4-mini
  namespace: default
spec:
  labelSelector:
    matchLabels:
      apps: phi-4-mini
  replicas: 1
  template:
    inference:
      preset:
        accessMode: public
        name: phi-4-mini-instruct
    resource:
      instanceType: Standard_NC24ads_A100_v4
EOF

In just a few seconds, the KEDA KAITO scaler automatically creates the scaledobject and hpa objects. After a few minutes, once the inference pod runs, the KEDA KAITO scaler begins scraping metric values from the inference pod. The system then marks the status of the scaledobject and hpa objects as ready.

# kubectl get scaledobject
NAME           SCALETARGETKIND                  SCALETARGETNAME   MIN   MAX   READY   ACTIVE    FALLBACK   PAUSED   TRIGGERS   AUTHENTICATIONS           AGE
phi-4-mini     kaito.sh/v1alpha1.InferenceSet   phi-4-mini        1     5     True    True     False      False    external   keda-kaito-scaler-creds   10m

# kubectl get hpa
NAME                    REFERENCE                   TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-phi-4-mini     InferenceSet/phi-4-mini     0/10 (avg)   1         5         1          11m

That's it! Your KAITO workloads will now automatically scale based on the average number of waiting inference requests(vllm:num_requests_waiting) across all workloads associated with InferenceSet/phi-4-mini in the cluster.

In the example below, if vllm:num_requests_waiting exceeds the threshold (10) for over 60 seconds, KEDA scales up by adding a new replica to InferenceSet/phi-4-mini. Conversely, if vllm:num_requests_waiting remains below the threshold (10) for more than 300 seconds, KEDA scales down the number of replicas.

Every 2.0s: kubectl describe hpa
Name:                                                     keda-hpa-phi-4-mini
Namespace:                                                default
Labels:                                                   app.kubernetes.io/managed-by=keda-operator
                                                          app.kubernetes.io/name=keda-hpa-phi-4-mini
                                                          app.kubernetes.io/part-of=phi-4-mini
                                                          app.kubernetes.io/version=2.18.1
                                                          scaledobject.keda.sh/name=phi-4-mini
Annotations:                                              scaledobject.kaito.sh/managed-by: keda-kaito-scaler
CreationTimestamp:                                        Tue, 09 Dec 2025 03:35:09 +0000
Reference:                                                InferenceSet/phi-4-mini
Metrics:                                                  ( current / target )
  "s0-vllm:num_requests_waiting" (target average value):  58 / 10
Min replicas:                                             1
Max replicas:                                             5
Behavior:
  Scale Up:
    Stabilization Window: 60 seconds
    Select Policy: Max
    Policies:
      - Type: Pods  Value: 1  Period: 300 seconds
  Scale Down:
    Stabilization Window: 300 seconds
    Select Policy: Max
    Policies:
      - Type: Pods  Value: 1  Period: 600 seconds
InferenceSet pods:  2 current / 2 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from external metric s0-vllm:num_requests_waiting(&Lab
elSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: phi-4-mini,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  True    ScaleUpLimit      the desired replica count is increasing faster than the maximum scale rate
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  33s   horizontal-pod-autoscaler  New size: 2; reason: external metric s0-vllm:num_requests_waiting(&LabelSelector{MatchLabels:ma
p[string]string{scaledobject.keda.sh/name: phi-4-mini,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

Summary​

KAITO's LLM inference service must scale inference instances dynamically to handle varying numbers of waiting requests: scaling up to prevent blocking when requests increase, and scaling down to optimize GPU usage when requests decrease. With the newly introduced InferenceSet CRD and KEDA KAITO scaler, configuring this setting in KAITO has become much simpler.

We're just getting started and would love your feedback. To learn more about KAITO inference workloads autoscaling and AI model deployment on AKS, check out the following links:

Resources​

• KEDA Auto-Scaler for inference workloads
• KAITO InferenceSet
• vLLM Production Metrics

Imagine this: your application is thriving, traffic spikes, and Kubernetes promises elasticity. You hit “scale,” expecting seamless provisioning - only to be greeted by errors like:

• SkuNotAvailable: The VM size (also referred to as VM SKU) you requested is not available.
• AllocationFailed: Azure can’t allocate the specific VM size with the constraints you requested in a particular region.
• Quota exceeded: Your subscription has hit its compute limits for a particular location or VM size.
• ZonalAllocationFailed: Azure can’t allocate the VM size with the constraints you requested in a particular zone.
• OverconstrainedAllocationRequest: Azure can’t allocate the specific VM size with the constraints you requested in a particular region.
• OverconstrainedZonalAllocationRequest: Azure can’t allocate the VM size with the constraints you requested in a particular zone.

For customers, these aren’t just error messages - they’re roadblocks. Pods remain pending, deployments stall, and SLAs tremble. Scaling isn’t just about adding nodes; it’s about finding capacity in a dynamic, multi-tenant cloud where demand often outpaces supply. In the case of quota gaps, usually users can increase their quotas in a particular location - but what about when a specific virtual machine size (also known as a "VM SKU") is simply unavailable? This can cause many challenges for users.

The Hidden Complexity Behind Capacity​

When using Kubernetes, every node pool is typically tied to a specific VM SKU, region, and zone, which can require some effort to update. In some scaling scenarios, high-demand VM SKUs can become unavailable in certain regions or zones. In this case, limiting node pools to a single VM size becomes a bottleneck that can result in capacity errors and an outage. You’re left juggling trade-offs: Do you overprovision SKUs “just in case” to ensure availability? Or risk underprovisioning and inability to scale? AKS offers two solutions that aim to address these capacity scaling challenges.

Breaking the Mold: Features That Change the Game​

Node Auto Provisioning (NAP): Smarter Scaling​

NAP offers a more intelligent scaling experience. Instead of you guessing the right VM size and precreating node pools, NAP uses pending pod resource requests to dynamically provision nodes that fit your workloads. Built on the open-source Karpenter project, NAP:

• Automates VM selection: Chooses optimal SKUs based on CPU, memory, and constraints
• Consolidates intelligently: Removes underutilized nodes, reducing cost
• Adapts in real time: Responds to pod pressure without manual intervention

Think of NAP as Kubernetes with foresight: provisioning what you need, when you need it, without the spreadsheet gymnastics. Without NAP, a single unavailable VM SKU can block scaling entirely. With NAP, AKS dynamically adapts to capacity fluctuations, ensuring workloads keep running on available VM sizes, even during regional/zonal shortages.

How NAP handles capacity errors​

When a requested VM SKU isn’t available due to regional or zonal capacity constraints, NAP doesn’t fail outright. Instead, NAP will automatically:

• Evaluate pending pod resource requirements (for example CPU, memory, GPU)
• Check if pending pods can fit on existing nodes
• Search across multiple VM SKUs within the allowed families defined in your NAP configuration files, which are custom resource definitions (CRDs) named NodePool and AKSNodeClass
• Provision an alternative SKU that meets the workload requirements and policy constraints

In the event that no VM sizes that match your requirements are available, NAP will only then send an error detailing that "No available SKU that meets your configuration definition is available." To mitigate this, make sure you reference a broad range of size options in the NAP configuration files (for example D-series, or multiple SKU families).

This flexibility is key to avoiding hard failures during scale-out. In the scenario where there are no SKUs available based on your configuration requirements, NAP will return an error stating that there were no available SKUs that matched your requirements. Typically this means the configuration requirements probably can be broader, to allow for more available VM sizes.

NAP vs Cluster Autoscaler​

In traditional Kubernetes, Cluster Autoscaler is the standard autoscaling experience that scales pre-existing same VM size node pools. The requirement for same size autoscaling is subject to availability limits of the selected VM sizes, and Cluster Autoscaler does not allow for changing the node pool's VM SKU. Should the specific SKU be unavailable, a capacity error occurs and your workloads are now stuck. To overcome this limitation, you may have to pre-create multiple node pools with different VM SKUs to avoid capacity exhaustion which introduces operational complexities. When zonal allocation constraints are also factored in, the complexity of traditional node pools increase further. In such scenarios, cluster autoscaler configurations may require one node pool per zone for each VM SKU to reliably scale.

NAP offers a new model based on individual virtual machines rather than node pools or Virtual Machine Scale Sets. NAP also provides versatility that can offer more capacity resilience and more cost optimization than traditional node pools using Cluster Autoscaler. Many of the capacity limitations and work-arounds are addressed with NAP.

For more on enabling NAP on your cluster, visit our NAP documentation as well as our docs on configuring the NodePool CRD and AKSNodeClass CRD.

Virtual Machine Node Pools: Flexibility at Scale​

Traditional node pools are rigid: one VM size per node pool. Virtual machine node pools break that limitation. With multi-SKU support, you can:

• Mix VM sizes within a single node pool for diverse workloads
• Fine-tune capacity without creating dozens of node pools
• Reduce operational overhead while improving resilience

Virtual machine node pools provide flexibility and versatility in capacity-constrained regions.

How virtual machine node pools handle capacity errors​

You can manually add or update alternative VM SKUs into your new or existing node pools. When a requested VM SKU isn't available due to a regional or zonal capacity constraint, you will receive a capacity error, and can resolve this error by simply adding and updating the VM SKUs in your node pools.

For more on enabling Virtual machine node pools on your cluster, visit our Virtual machine node pools documentation.

Quick Guidance: When to Use What​

Generally, NAP and virtual machine node pools are mutually exclusive options. You can use NAP to create standalone VMs that NAP manages instead of traditional node pools, allowing for mixed SKU autoscaling. Virtual machine node pools use traditional node pools and allow for mixed SKU manual scaling.

• (Recommended) Choose NAP for dynamic environments or specific SKU selection where manual SKU planning is impractical.
• Choose virtual machine node pools when you need fine-tuned control with exact VM SKUs for compliance, predictable performance, or cost modeling

Avoid NAP if you require strict SKU governance or have regulatory constraints that cannot allow for dynamic autoscaling. Avoid VM node pools if you want full automation without manual profiles.

Best Practices for Resilience​

To maximize NAP's ability to handle capacity errors:

• Define broad SKU families (e.g., D, E) in your NodePool requirements
• Avoid overly restrictive affinity rules. Visit our node selector and affinity best practices documentation for more details
• Enable multiple NodePools with different priorities for fallback. Visit our NAP Node Pool documentation to learn more

To maximize virtual machine node pool's ability to adapt to capacity errors:

• Be clear on a list of VM SKUs that can tolerate your workloads. Visit our Azure VM Sizes documentation for more details
• Create virtual machine node pools to offer resiliency to your workloads. Visit our virtual machine node pool documentation on how to add a mixed SKU node pool

Getting Started with Node Auto Provisioning​

Before you begin, visit our NAP documentation on minimum cluster requirements.

Create a new NAP-managed AKS cluster​

The following command creates a new NAP-managed AKS cluster by setting the --node-provisioning-mode field to Auto. This command also sets the network configuration to the recommended Azure CNI Overlay with a Cilium dataplane (optional). View our NAP networking documentation for more on supported CNI options.

az aks create --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --node-provisioning-mode Auto --network-plugin azure --network-plugin-mode overlay --network-dataplane cilium

Update an existing cluster to be a NAP-managed cluster​

The following command updates an existing cluster to enable NAP:

az aks update --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --node-provisioning-mode Auto

Configure NAP CustomResourceDefinitions​

NAP uses CustomResourceDefinitions (CRDs) and your application deployment file requirements for its decision-making. The Karpenter controller takes this information and determines which virtual machines to provision and schedule your workloads to. Karpenter CRD types include:

• NodePool - for setting rules around the range of VM sizes, capacity type (spot vs. on-demand), compute architecture, availability zones, etc
• AKSNodeClass - for setting rules around certain Azure specific settings such as more detailed networking (virtual networks) setup, node image family type, operating system configurations, and other resource-related definitions

Visit our NAP NodePool Documentation and NAP AKSNodeClass documentation for more on configuring these files.

Getting started with virtual machine node pools​

Create a new AKS cluster with virtual machine node pools​

The following example creates a new cluster named myAKSCluster with a virtual machine node pool containing two nodes with size "Standard_D4s_v3", and sets the Kubernetes version to 1.31.0:

az aks create --resource-group myResourceGroup --name myAKSCluster \
  --vm-set-type "VirtualMachines" --vm-sizes "Standard_D4s_v3" \
  --node-count 2 --kubernetes-version 1.31.0

Add a new virtual machine node pool to an existing cluster​

The following example adds a virtual machine node pool named myvmpool to the myAKSCluster cluster. The node pool creates a ManualScaleProfile with --vm-sizes set to Standard_D4s_v3 and a --node-count of 3:

az aks nodepool add --resource-group myResourceGroup --cluster-name myAKSCluster --name myvmpool --vm-set-type "VirtualMachines" --vm-sizes "Standard_D4s_v3" --node-count 3

With virtual machine node pools you can also perform some of the following commands:

• Add multiple VM sizes in an existing or new node pool
• Update VM sizes in an existing node pool
• Single-SKU autoscaling (public preview)
• Delete VM sizes in an existing node pool

Visit our virtual machine node pools documentation for more info.

Upcoming experiences on the AKS roadmap​

• NAP: Expect deeper integration with cost optimization tools and advanced disruption policies for even smarter consolidation.
• Virtual machine node pools: Multi-SKU autoscaling (general availability) is on the horizon, reducing manual configuration and enabling adaptive scaling across mixed SKUs.

Next steps​

Ready to get started?

1. Try one of these features now: Follow the Enable Node Auto Provisioning steps or create a virtual machine node pool.
2. Share feedback: Open issues or ideas in AKS GitHub Issues.
3. Join the community: Subscribe to the AKS Community YouTube and follow @theakscommunity on X.

In this shared registry model, traditional Azure role-based access control (RBAC) forces an all-or-nothing choice: either grant broad registry-wide permissions or manage separate registries per team. Neither approach aligns with least-privilege principles or modern zero trust architectures.

Microsoft Entra attribute-based access control (ABAC) for Azure Container Registry solves this challenge. ABAC augments Azure RBAC with fine-grained conditions, enabling platform teams to scope permissions precisely to specific repositories or namespaces within a shared registry. CI/CD pipelines and Azure Kubernetes Service (AKS) clusters can now access only their authorized repositories, eliminating overprivileged authorization while maintaining operational simplicity.

How ABAC works in ACR​

Attribute-based access control (ABAC) extends the traditional RBAC model by introducing attributes on top of actions and data actions. While RBAC defines who (identity) can perform what permissions (Entra actions and data actions in a role definition) on which resource scope (subscription, resource group, or registry), ABAC adds a fourth dimension: attributes. Attributes—such as repository name patterns—are properties of the resource being accessed. ABAC uses conditions in role assignments to evaluate these attributes and determine whether access should be granted.

For ACR, this means that while the scope of a role assignment can be at the subscription, resource group, or registry level, ABAC conditions evaluate repository-level attributes like repositories:name to determine whether a specific permission (such as repositories/content/read) is permitted. This allows administrators to scope permissions to specific repositories or namespace prefixes within a single registry, without needing separate registries or overprivileged access.

ACR registries support a permissions mode called "RBAC Registry + ABAC Repository Permissions" that makes them ABAC-enabled. Once configured, registry administrators add ABAC conditions to standard Azure RBAC role assignments, scoping permissions to specific repositories or namespace prefixes. This enables:

• CI/CD pipelines to push images only to their approved namespaces and repositories
• AKS clusters, Azure Container Apps, and Azure Container Instances to pull only from authorized repositories
• Microsoft Entra identities to enforce permission boundaries through standard role assignments

Enabling ABAC on ACR​

By default, ACR registries use the "RBAC Registry Permissions" authorization mode, which provides traditional registry-scoped access control. ABAC can be enabled on all new and existing ACR registries across all SKUs by changing the authorization mode to "RBAC Registry + ABAC Repository Permissions", either during registry creation or configured on existing registries. You can assign RBAC roles with optional ABAC conditions to users, groups, service principals, and managed identities used by resources such as Azure Kubernetes Service (AKS), Azure Container Apps (ACA), and Azure Container Instances (ACI) when pulling images from ACR.

Here is the Azure Portal experience for enabling ABAC on a new ACR during creation:

Here is the Azure Portal experience for enabling ABAC on an existing ACR:

ABAC can also be enabled on ACR registries through Azure Resource Manager (ARM), Bicep files, Terraform templates, and Azure CLI.

ABAC-enabled built-in roles​

Once a registry is ABAC-enabled (configured to "RBAC Registry + ABAC Repository Permissions"), registry admins can use these ABAC-enabled built-in roles to grant repository-scoped permissions:

• Container Registry Repository Reader: grants image pull and metadata read permissions, including permissions for HEAD requests, GET manifest requests, GET layer blob requests, tag resolution, and discovering OCI referrers.
• Container Registry Repository Writer: grants Repository Reader permissions, as well as image and tag push permissions.
• Container Registry Repository Contributor: grants Repository Reader and Repository Writer permissions, as well as image and tag delete permissions.

Note that these roles do not grant repository list permissions.

• The separate Container Registry Repository Catalog Lister must be assigned to grant repository list permissions.
• The Container Registry Repository Catalog Lister role does not support ABAC conditions in role assignments; assigning this role grants permissions to list all repositories in a registry.

Assigning roles with ABAC conditions​

After enabling ABAC on your registry, you can create role assignments with conditions that scope permissions to specific repositories. Here's an example using Azure CLI to grant a managed identity read access to repositories matching the myapp/* namespace:

az role assignment create \
  --assignee "<managed-identity-principal-id>" \
  --role "Container Registry Repository Reader" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ContainerRegistry/registries/<registry-name>" \
  --condition "((!(ActionMatches{'Microsoft.ContainerRegistry/registries/repositories/content/read'})) OR (@Resource[Microsoft.ContainerRegistry/registries/repositories:name] StringStartsWith 'myapp/'))" \
  --condition-version "2.0"

In the Azure Portal, navigate to your registry's Access control (IAM) blade, select Add role assignment, choose an ABAC-enabled role, and configure conditions in the Conditions tab to specify repository name patterns.

Important role behavior changes in ABAC mode​

• Legacy data-plane roles such as AcrPull, AcrPush, and AcrDelete are not honored in ABAC-enabled registries and should not be used. For ABAC-enabled registries, use the ABAC-enabled built-in roles listed above.
• Broad roles like Owner, Contributor, and Reader previously granted full control plane and data plane permissions. This is typically an overprivileged role assignment. In ABAC-enabled registries, these broad roles will only grant control plane permissions to the registry. Owner, Contributor, and Reader will no longer grant data plane permissions, such as image push, pull, delete, or repository list permissions.
• In ABAC-enabled registries, ACR Tasks, Quick Tasks, Quick Builds, and Quick Runs no longer have default data plane access to source registries. This prevents inadvertent security leaks and broad permissions grants to ACR Tasks. To grant an ACR Task permissions to a source ACR registry, assign the ABAC-enabled roles above to the calling identity of the Task or Task Run as needed.

Next steps​

Start using ABAC repository permissions in ACR to enforce least-privilege artifact push, pull, and delete boundaries across your CI/CD systems and container image workloads. This model is now the recommended approach for multi-tenant platform engineering patterns to secure container registry deployments.

To get started, follow the step-by-step guides in the official ACR ABAC documentation.

In our previous post, we demonstrated the power of the Azure ND GB200-v6 VMs accelerated by NVIDIA GB200 NVL72, achieving a staggering 1.2M tokens per second across 10 nodes using NVIDIA Dynamo. Today, we're shifting focus from raw throughput to developer velocity and operational efficiency.

We will explore how the Dynamo Planner and Dynamo Profiler remove the guesswork from performance tuning on AKS.

The Challenge: Balancing the "Rate Matching" Equation​

Disaggregated serving separates the prefill phase (when the model first processes the entire input sequence at once) and decode phase (when the model starts sequentially generating output tokens) of inference across distinct GPU nodes. This allows each phase to be independently optimized with custom GPU counts and model parallelism configurations.

One of the main challenges in disaggregated serving is rate matching: determining the right GPU allocation between prefill and decode stages to meet a specific Service Level Objective (SLO). If you miscalculate the GPU ratio between these stages, you face two "silent killers" of performance:

• Over-provisioned Prefill: Your prompt processing is fast, but requests bottleneck at the generation stage. This spikes Inter-Token Latency (ITL) and leaves expensive compute nodes idle.
• Under-provisioned Prefill: Your decode GPUs sit starved for data. This drives up Time-To-First-Token (TTFT) and inflates your Total Cost of Ownership (TCO).

Beyond rate matching, developers must also optimize model parallelism parameters (data, tensor, and expert parallelism) to maintain high "Goodput" (the fraction of time and resources where the model is learning or producing correct results, instead of waiting or doing extra work).

Exploring these configurations manually is technically challenging, time-consuming and often results in suboptimal resource utilization.

Dynamic Traffic: The Move to SLO-Driven Scaling​

Static configurations are brittle. In production, traffic is rarely uniform:

• Volatile Request Volume: Traditional Horizontal Pod Autoscalers (HPA) are too slow for LLM jitters.
• Shifting Sequence Patterns: If your workload shifts from short chat queries (low input sequence length (ISL)) to long-context document analysis (high ISL), a static disaggregated split becomes suboptimal instantly (resulting in overworked prefill GPUs and idle decode GPUs).

NVIDIA Dynamo addresses these gaps through two integrated components: the Planner Profiler and the SLO-based Planner.

Let’s see it through an example application scenario​

Consider a mission-critical AI workload running on AKS: an airline’s automated rerouting system during a widespread delay. This use case is a 'stress test' for inference: it is subject to massive, sudden bursts in traffic and highly variable request patterns, such as a mix of short status queries and long-context itinerary processing. To prevent latency spikes during these peaks, the underlying system requires the precise orchestration offered by a disaggregated architecture.

Using the Qwen3-32B-FP8 model, we can deploy an Airline Assistant with strict SLA targets: TTFT ≤ 500ms and ITL (Inter-Token Latency) ≤ 30ms.

During normal operations, passengers ask short queries like "What's my flight status?" But when a major weather system causes flight cancellations, passengers flood the app with complex rerouting requests—long-context queries (~4,000 tokens) requiring detailed itinerary responses (~500 tokens). This sudden surge of 200 concurrent users is exactly the kind of real-world spike that breaks static configurations.

To build a truly efficient disaggregated AI inference system, you need to transition from manual "guess-and-check" configurations to an automated, SLO-driven approach. The core of this automation lies in two distinct but deeply integrated components: the Dynamo Planner profiler and the Dynamo Planner.

The first step in building your system is determining the "Golden Ratio" of GPUs: how many should handle prefill versus decode, and what tensor parallelism (TP) levels each should use.

The Architect: Dynamo Profiler​

The Dynamo Planner profiler is your pre-deployment simulation engine. Instead of burning GPU hours testing every possible configuration, you define your requirements in a DynamoGraphDeploymentRequest (DGDR) manifest. The profiler then executes an automated "sweep" of the search space:

• Parallelization Mapping: It tests different TP sizes for both prefill and decode stages to find the lowest TTFT and ITL.
• Hardware Simulation: Using the AI Configurator (AIC) mode, the profiler can simulate performance in just 20–30 seconds based on pre-measured performance data, allowing for rapid iteration before you ever touch a physical GPU.
• Resulting Recommendation: The output is a highly tuned configuration that maximizes "Goodput", the maximum throughput achievable while staying strictly within your latency bounds.

Ultimately, the app developers and AI engineers reduce their time spent on testing different system setups, and can focus on their airline passengers’ needs.

The Pilot: Dynamo Planner​

Once your system is deployed, static configurations can't handle the "jitter" of real-world traffic. This is where the Dynamo Planner takes over as a runtime orchestration engine.

Unlike a traditional load balancer, the Dynamo Planner is LLM-aware. It continuously monitors the live state of your cluster, specifically looking at:

• KV Cache Load: It monitors memory utilization in the decode pool.
• Prefill Queue Depth: It tracks how many prompts are waiting to be processed.

Using the performance bounds identified earlier by the profiler (i.e. TTFT ≤ 500ms and ITL ≤ 30ms) the Planner proactively scales the number of prefill and decode workers up or down. For example, if a sudden burst of long-context itinerary queries floods the system, the Planner detects the spike in the prefill queue and shifts available GPU resources to the prefill pool before your TTFT violates its SLO.

Seeing it in Action​

In our airline scenario, the system starts with 1 prefill worker and 1 decode worker. When the passenger surge hits, the Planner's 60-second adjustment interval detects the SLA violations:

Prefill calculation: 138.55 (p_thpt) / 4838.61 (p_engine_cap) = 1 (num_p)
Decode calculation: 27.27 (d_thpt) / 19381.08 (d_engine_cap) = 1 (num_d)

As traffic spikes to 200 concurrent passengers, the Planner recalculates:

Prefill calculation: 16177.75 (p_thpt) / 8578.39 (p_engine_cap) = 2 (num_p)
Decode calculation: 400.00 (d_thpt) / 3354.30 (d_engine_cap) = 1 (num_d)
Predicted number of engine replicas: prefill=2, decode=1
Updating prefill component VllmPrefillWorker to desired replica count 2

See the Dynamo SLA Planner in action as it automatically scales the Airline Assistant during a traffic surge. The Planner automatically scales to 2 prefill workers while keeping 1 decode worker (the optimal configuration to handle the surge while maintaining SLA targets). Within minutes, the new worker is online and passengers are getting their rerouting options without frustrating delays.

Now, you can try this yourself by running the NVIDIA Dynamo Planner Profiler to capture burst and request behavior, then using the SLO-based Planner to translate latency targets into placement and scaling decisions on your AKS cluster. Setting it up in this order - profile under stress, define SLOs, and let the planner orchestrate your disaggregated inference system to handle sudden traffic spikes without latency spikes.

After deploying Dynamo by following these instructions, get hands on with the Qwen3-32B-FP8 model using the example in AKS Dynamo Part 2 sample.

Conclusion: Inference Without the Infrastructure Burden​

The shift toward disaggregated serving is a necessity for the next generation of reasoning-heavy and long-context LLMs. However, as we have seen, the complexity of manually tuning these distributed systems on Kubernetes can quickly become a bottleneck for even the most experienced AI teams.

By utilizing the NVIDIA Dynamo Planner Profiler, developers can move from educated guessing to data-driven certainty, modeling performance in seconds rather than days. When paired with the Dynamo Planner, this static optimization becomes a dynamic, SLO-aware reality on AKS, capable of weathering the unpredictable traffic spikes of production environments.

Ultimately, this suite transforms your inference stack from a series of fragile configurations into a resilient, self-optimizing engine. For the AI engineer, this means less time spent managing hardware limits and configuring system scalability and more time spent delivering the high-quality, interactive experiences that your users (and your passengers) expect.

The challenge​

When you create an AKS Automatic cluster, Azure enables several production-ready defaults, including:

• Azure RBAC for Kubernetes authorization
• Disabled local accounts (no static kubeconfig credentials)
• Workload Identity authentication

These defaults strengthen security, but they also mean the typical Helm provider configuration that relies on a kubeconfig file won't work. Instead, you need to configure the Helm provider to use token-based authentication.

Prerequisites​

Before you begin, ensure you have:

• Terraform 1.14 or later
• Azure CLI 2.81 or later, installed and authenticated
• kubectl v1.34 or later
• kubelogin v0.2.13 or later
• Helm v3 or later
• An Azure subscription with permissions to create AKS clusters

Deploy AKS Automatic with AzApi provider​

The AzApi provider is a lightweight Terraform provider that allows you to deploy Azure resources using the Azure Resource Manager (ARM) API. In most cases, it's simpler to use the AzureRM provider but when you're looking to deploy new Azure services that aren't yet supported by the AzureRM provider, AzApi is a great alternative.

Create a new directory and add a Terraform configuration file (for example, main.tf) with the following code to deploy an AKS Automatic cluster:

terraform {
  required_version = ">= 1.14, < 2.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 4.0.0, < 5.0.0"
    }
    azapi = {
      source  = "azure/azapi"
      version = ">= 2.8.0, < 3.0.0"
    }
    random = {
      source  = "hashicorp/random"
      version = ">= 3.5.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 3.0.0, < 4.0.0"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

data "azurerm_client_config" "current" {}

resource "random_pet" "this" {
  separator = ""
}

resource "azurerm_resource_group" "this" {
  location = "swedencentral"
  name     = "rg-${random_pet.this.id}"
}

resource "azapi_resource" "this" {
  type                      = "Microsoft.ContainerService/managedClusters@2025-10-01"
  parent_id                 = azurerm_resource_group.this.id
  location                  = azurerm_resource_group.this.location
  name                      = "aks-${random_pet.this.id}"
  schema_validation_enabled = false # Use when the azapi provider's local schema validation doesn't yet support this API version

  body = {
    identity = {
      type = "SystemAssigned"
    },
    properties = {
      agentPoolProfiles = [
        {
          name  = "systempool"
          mode  = "System"
          count = 3
        }
      ]
    }
    sku = {
      name = "Automatic"
      tier = "Standard"
    }
  }
}

resource "azurerm_role_assignment" "this" {
  scope                = azapi_resource.this.id
  principal_id         = data.azurerm_client_config.current.object_id
  role_definition_name = "Azure Kubernetes Service RBAC Cluster Admin"
}

This minimal configuration creates an AKS Automatic cluster and assigns the current user the Azure Kubernetes Service RBAC Cluster Admin role.

Run the following commands to deploy the cluster:

terraform init
terraform apply

After 7 to 10 minutes, your AKS cluster is ready.

Helm provider authentication options​

The Helm provider allows you to authenticate to a Kubernetes cluster in several ways:

1. Using a kubeconfig file: This option doesn't work because local accounts are disabled in AKS Automatic clusters.
2. Supplying credentials directly: This option is possible, but with a twist—you need to obtain a bearer token first and then supply it to the provider.
3. Using the exec plugin: This option calls an external program to obtain short-lived credentials. It uses the client-go credential plugin mechanism built into kubectl and the Kubernetes client libraries.

With that context, let's explore the two viable options for configuring the Helm provider.

Add the following to the bottom of your main.tf file:

data "azurerm_kubernetes_cluster" "this" {
  name                = azapi_resource.this.name
  resource_group_name = azurerm_resource_group.this.name
}

Option 1: Configure the Helm provider with Azure bearer token authentication​

The Helm provider's kubernetes block supports a token argument that lets you supply a bearer token directly for authentication.

You can obtain short-lived access tokens using the Azure CLI. If you're authenticated with az login, you can get a token for the AKS resource like this:

az account get-access-token --resource 6dae42f8-4368-4678-94ff-3960e28e3630

Combining the above command with the external data source in Terraform allows you to retrieve the token dynamically and use it in the Helm provider configuration.

Add the following to your main.tf file:

data "external" "this" {
  program = ["bash", "-c", "az account get-access-token --resource 6dae42f8-4368-4678-94ff-3960e28e3630 --query '{token: accessToken}' -o json"]
}

You can now reference the token in the Helm provider configuration like this:

provider "helm" {
  kubernetes = {
    host                   = data.azurerm_kubernetes_cluster.this.kube_config.0.host
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config.0.cluster_ca_certificate)
    token                  = data.external.this.result.token
  }
}

This configuration uses the host and cluster CA certificate from the AKS cluster data source and gains access to the cluster using the bearer token from the external data source.

For quick local demos, this approach is convenient. For CI/CD pipelines or service principal authentication, the next option is more flexible.

Option 2: Configure the Helm provider to use the exec plugin with kubelogin​

The Helm provider also supports using the exec plugin mechanism to obtain credentials dynamically. This approach is more flexible and works well with various authentication methods supported by kubelogin which is a Kubernetes client-go credential plugin for Azure.

To use this approach, add the following to your main.tf:

provider "helm" {
  kubernetes = {
    host                   = data.azurerm_kubernetes_cluster.this.kube_config.0.host
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config.0.cluster_ca_certificate)

    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "kubelogin"
      args = [
        "get-token",
        "--login",
        "azurecli",
        "--server-id",
        "6dae42f8-4368-4678-94ff-3960e28e3630"
      ]
    }
  }
}

This configuration uses the kubelogin tool to obtain an access token based on your existing Azure CLI authentication context. Therefore, you must ensure it is installed and accessible in your system's PATH.

provider "helm" {
  kubernetes = {
    host                   = data.azurerm_kubernetes_cluster.this.kube_config.0.host
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config.0.cluster_ca_certificate)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "kubelogin"             # Make sure kubelogin is installed and accessible in PATH
      args = [
        "get-token",
        "--login",
        "spn",
        "--environment",
        "AzurePublicCloud",                 # Adjust if using a different cloud
        "--server-id",
        "6dae42f8-4368-4678-94ff-3960e28e3630",
        "--client-id",
        var.service_principal_client_id,    # Replace with your service principal client ID
        "--tenant-id",
        var.service_principal_tenant_id     # Replace with your service principal tenant ID
      ]
      env = {
        AAD_SERVICE_PRINCIPAL_CLIENT_SECRET = var.service_principal_client_secret
      }
    }
  }
}

Deploy an application with Helm​

Now you can use the Helm provider to deploy applications. Add the following to your main.tf file to deploy the AKS Store Demo application Helm chart:

resource "helm_release" "example" {
  name       = "aks-store-demo"
  repository = "https://azure-samples.github.io/aks-store-demo"
  chart      = "aks-store-demo-chart"
  version    = "1.5.0"
}

Run the following commands and you'll see the Helm provider uses the configured authentication method to connect to the AKS Automatic cluster and deploy the AKS Store Demo application.

terraform init -upgrade
terraform apply

If all goes well, you should see output indicating the release was deployed successfully 🚀

Conclusion​

AKS Automatic provides production-ready defaults that improve security but require a different authentication approach for automation tools like Helm or Kubernetes providers for Terraform. By using the exec plugin with kubelogin or bearer token-based approaches, you can seamlessly integrate Helm deployments into your Terraform workflow while maintaining the security benefits of Azure RBAC.

Resources​

• AKS Automatic documentation
• AzApi provider documentation
• Terraform Helm Provider
• kubelogin documentation
• Azure Kubernetes Service RBAC roles

Azure Kubernetes Service (AKS) is proud to be among the first platforms certified for Kubernetes AI Conformance, demonstrating our commitment to providing customers with a verified, standardized platform for running AI workloads.

What is the Kubernetes AI Conformance Program?​

The Kubernetes AI Conformance Program defines a standard set of capabilities, APIs, and configurations that a Kubernetes cluster must offer to reliably and efficiently run AI and ML workloads. Building on CNCF's successful Certified Kubernetes Conformance Program, which brought together more than 100 certified distributions and platforms, this new initiative applies the same proven model to AI infrastructure.

The program is developed in the open by the Kubernetes community.

Why AI Conformance matters​

Running AI workloads on Kubernetes introduces unique challenges. Teams often struggle with GPU driver compatibility, specialized scheduling requirements for distributed training, and the complexity of exposing inference endpoints at scale. Without a common baseline, organizations risk building on fragmented, vendor-specific implementations that limit flexibility and portability.

According to Linux Foundation Research on Sovereign AI, 82% of organizations are already building custom AI solutions, and 58% use Kubernetes to support those workloads. With 90% of enterprises identifying open-source software as critical to their AI strategies, the risk of fragmentation and inconsistent performance is rising. The Kubernetes AI Conformance Program responds directly to this need.

The AI Conformance Program addresses these challenges by establishing a verified set of capabilities that every conformant platform must support. When you deploy on AKS, a certified AI-conformant platform, you benefit from:

• Predictable scaling: Your AI workloads scale consistently using standardized APIs and autoscaling behaviors
• Hardware optimization: GPU and accelerator resources are managed through proven Kubernetes primitives, maximizing utilization
• Workload mobility: Applications built for one conformant platform work on any other, reducing vendor lock-in
• Ecosystem compatibility: Popular ML frameworks, operators, and tools function reliably because they target a known, tested baseline

Key requirements for AI Conformance​

The Kubernetes AI Conformance program maintains a versioned specification of requirements for each Kubernetes release starting with v1.33. Each requirement goes through a graduation process, starting as a SHOULD recommendation and may eventually become a MUST for certification.

Here are the requirements for Kubernetes v1.34 and how AKS meets each requirement:

Accelerators​

Dynamic Resource Allocation (DRA)​

Traditional resource requests in Kubernetes use simple numeric counts (for example, nvidia.com/gpu: 1). DRA introduces a more flexible model where workloads can specify device characteristics, request specific GPU models, or express preferences about memory and compute capabilities. DRA APIs are enabled by default in Kubernetes 1.34 on AKS. For a deep dive into how DRA works and how to use it with GPU drivers, see our blog post on DRA: Devices and Drivers on Kubernetes.

Networking​

Gateway API for AI inference​

Inference services often need sophisticated traffic routing: sending a percentage of requests to a new model version, routing based on request headers (including OpenAI protocol headers), or implementing canary deployments. The Kubernetes Gateway API introduces improved routing capabilities for a more flexible architecture for traffic management. AKS supports Gateway API through both the Istio-based service mesh add-on and Application Gateway for Containers.

Scheduling and orchestration​

Gang scheduling​

Distributed training jobs often require multiple pods to start simultaneously. If only some pods in a training job get scheduled, the others wait indefinitely, wasting resources. Gang scheduling solves this by treating a group of pods as a single unit: either all pods get scheduled together, or none do. You can run Kueue on AKS to enable gang scheduling. To get started, refer to the guidance on deploying batch jobs with Kueue.

Cluster autoscaling​

GPU nodes are expensive. The cluster autoscaler must intelligently provision GPU nodes when training or inference pods are pending, and scale them down during idle periods. On AKS, you can combine the cluster autoscaler with KEDA and NVIDIA DCGM metrics to automatically scale GPU node pools based on real-time utilization. You can configure GPU node pools to scale down to zero when no workloads are running, minimizing costs.

Pod autoscaling​

Inference workloads need to scale pod replicas based on demand. The Horizontal Pod Autoscaler (HPA) must work correctly with GPU-enabled pods and support custom metrics relevant to AI workloads. On AKS, you can use KEDA with NVIDIA DCGM metrics to scale pods based on GPU metrics like DCGM_FI_DEV_GPU_UTIL (GPU utilization percentage). This enables scaling decisions based on actual GPU usage rather than just CPU or memory.

Observability​

Accelerator performance metrics​

Observability is critical for AI workloads. Platforms must expose GPU utilization, memory consumption, temperature, power draw, and other accelerator metrics through standard endpoints. You can deploy the NVIDIA DCGM exporter on AKS to expose GPU metrics in Prometheus format, and use GPU health monitoring for proactive issue detection with Node Problem Detector (NPD) in your AI pipelines.

AI service metrics​

Platforms must provide a monitoring system capable of discovering and collecting metrics from workloads that expose them in standard formats like Prometheus. AKS integrates with Azure Monitor for containers and provides Container Insights GPU monitoring for observability needs in your environment. For AI inference workloads, you can monitor and visualize vLLM inference metrics with Azure Managed Prometheus and Azure Managed Grafana when using the AI toolchain operator (KAITO) add-on.

Security​

Secure accelerator access​

Access to accelerators from within containers must be properly isolated and mediated by the Kubernetes resource management framework (device plugin or DRA) and container runtime, preventing unauthorized access or interference between workloads. See the AKS secure accelerator access evidence for implementation details.

Operators​

AI operator support​

Modern AI platforms rely on Kubernetes operators to manage complex resources like training jobs, model servers, and distributed training coordinators. AKS supports the AI Toolchain Operator (KAITO) to run inferencing, fine-tuning, and retrieval augmented generation (RAG). You can also install and run operators with custom resource definitions, such as Ray on AKS.

Creating an AI-conformant AKS cluster​

To create an AI-conformant AKS cluster, be sure to choose a certified Kubernetes version with the appropriate features enabled.

Step 1. Create an AKS cluster​

AKS achieved AI Conformance certification starting with Kubernetes 1.34. Create an AKS cluster running Kubernetes 1.34 or later:

az aks create \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --enable-azure-monitor-metrics \
  --kubernetes-version 1.34.0

Step 2. Add a GPU node pool​

For GPU-accelerated workloads, add a fully managed GPU node pool (preview) with GPU-enabled VMs. For a select set of NVIDIA GPU SKUs, AKS automatically installs the GPU driver, device plugin, and DCGM metrics exporter.

First, register the feature flag:

az feature register --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview

Then add the GPU node pool with the EnableManagedGPUExperience tag:

az aks nodepool add \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --name gpunp \
  --node-count 1 \
  --node-vm-size Standard_NC40ads_H100_v5 \
  --node-taints sku=gpu:NoSchedule \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 3 \
  --tags EnableManagedGPUExperience=true

Step 3. Enable the Istio service mesh with Gateway API (optional)​

If your AI workloads require advanced traffic management capabilities, you can use the Istio service mesh add-on with the Gateway API (preview).

First, install the Azure CLI preview extension and register the feature flag:

# Install the aks-preview extension
az extension add --name aks-preview

# Register the Gateway API feature flag
az feature register --namespace "Microsoft.ContainerService" --name "ManagedGatewayAPIPreview"

Then enable the Istio service mesh and Gateway API on your cluster:

az aks mesh enable \
  --resource-group <resource-group> \
  --name <cluster-name>

az aks update \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --enable-gateway-api

Step 4. Enable Prometheus metrics (optional)​

If you need to collect metrics about your applications and infrastructure, you can optionally install the metrics add-on that scrapes Prometheus metrics.

az aks update \
  --resource-group <resource-group> \
  --name <cluster-name> \
  --enable-azure-monitor-metrics

Your cluster now meets the requirements for Kubernetes AI Conformance. You can now optionally install and run Kueue to enable gang scheduling and deploy batch jobs.

Conclusion​

The Kubernetes AI Conformance Program represents an important step forward for the AI ecosystem, aligning the community around shared standards that make deploying AI at scale more consistent and reliable. By using an AI-conformant platform like AKS, you can build AI applications that are production-ready, portable, and efficient without reinventing infrastructure for every deployment. For more information about the program and how to get involved, check out the CNCF Kubernetes AI Conformance Repository.

AKS's certification demonstrates Microsoft's commitment to open standards and ensures that your AI workloads can run reliably on a verified platform. Start building your AI-conformant AKS cluster today and take advantage of the growing ecosystem of compatible tools and frameworks.

Resources​

• CNCF Kubernetes AI Conformance Repository
• AKS AI/ML Documentation
• CNCF Kubernetes AI Conformance Announcement

AKS Automatic simplifies this by enabling teams to ship applications with production-grade defaults from day one. With managed system node pools (preview), AKS takes this further. The system pool is now fully managed by Microsoft. Core cluster components run on Microsoft-owned infrastructure, so you no longer provision, patch, or scale system nodes. You focus on your apps while AKS handles the operational overhead of keeping the cluster healthy.

Automatic clusters with managed system node pools also introduce the Pod readiness Service Level Agreement (SLA). Beyond API server uptime, AKS now guarantees your pods reach readiness and serve users.

Why it matters​

• Reduced operational overhead: AKS handles provisioning, patching, upgrades, and scaling for the system pool, so you spend less time on infrastructure maintenance.
• Managed add-on hosting at lower cost: Core services like Azure Monitor collectors, CoreDNS, KEDA, VPA, Konnectivity, Eraser, and Metrics Server run on Microsoft-owned infrastructure. Some add-ons and DaemonSets still run on nodes in your subscription.
• Built-in security policies: Deployment Safeguards enforce pod security standards, restrict access to platform namespaces, and block risky configurations by default.
• Automatic upgrades: AKS keeps platform components current, reducing the risk of running outdated or vulnerable system software.
• Pod readiness SLA: A financially backed guarantee that your pods reach readiness and serve traffic, not just that your cluster is healthy. Refer to the SLA for details.

Components running on managed system node pools​

AKS manages the following platform components on the managed system node pool. You don't need to provision capacity for these services.

Other add-ons and extensions, outside of that list, run on aks-system-surge nodes, with scaling handled by Node Auto-Provisioning (NAP). DaemonSets run on both managed system node pools and nodes in your subscription.

How managed system node pools differ from traditional system node pools​

Guardrails for security and reliability​

Security misconfigurations are a leading cause of container breaches. AKS Automatic addresses this by enforcing Deployment Safeguards that validate every workload against the Kubernetes Pod Security Standards before it reaches your cluster. Baseline policies block dangerous privilege escalations while restricted policies enforce maximum hardening. Compliance flows into Azure Policy dashboards automatically.

These policies also improve workload reliability. Resource limits prevent runaway containers from starving neighbors. Health probes ensure traffic reaches only healthy pods. Anti-affinity rules spread replicas across failure domains. PodDisruptionBudget validation keeps node maintenance on schedule.

Since AKS manages the system node pool on your behalf, additional restrictions protect platform stability. User workloads cannot run on the managed system node pool and all create, update, and delete operations on managed system pool resources are denied since Microsoft hosts the system node pool outside of your subscription, as are pod exec, attach, and kubectl debug operations.

Preventing container escapes: Blocking privileged containers, host namespaces, host ports, and hostPath volumes for alignment with security best practices.

Reducing attack surface: Restricting Linux capabilities to a minimal set means processes run with only the permissions they need. Fewer capabilities translate directly to fewer exploitation opportunities.

Enforcing least privilege: Requiring containers to run as non-root and disabling privilege escalation limits the blast radius of any vulnerability.

Maintaining kernel protections: Seccomp, AppArmor, and SELinux profiles filter system calls and confine container behavior. Policies ensure these protections stay active.

Enabling safe cluster operations: Limiting sysctls to safe parameters and protecting node objects ensures platform components run undisturbed and node drains proceed smoothly.

For detailed specifications, see the Deployment Safeguards documentation.

Pod Readiness SLA for AKS Automatic​

Uptime means more than a healthy control plane; it means your applications are actually serving users. The Pod Readiness SLA guarantees that pods reach readiness targets, closing the gap between "the cluster is healthy" and "my app is ready."

• Faster recovery during failures: Node failures and scale events trigger remediation so pods return to a ready state within defined thresholds.
• Predictable reliability: Availability planning aligns with measurable guarantees instead of best-effort behavior.
• Reduced operational overhead: Platform automation handles remediation, eliminating manual firefighting during disruptions.
• Business continuity at scale: Mission-critical services experience minimal disruption even during infrastructure events.

Pricing​

AKS Automatic pricing includes a fixed monthly cluster fee and per-vCPU charges on top of standard VM compute costs. This pricing includes financially backed SLAs for both API server uptime and pod readiness. For current rates and a full breakdown by VM category, see the Azure Kubernetes Service pricing page.

Getting started​

Prerequisites​

• Azure CLI 2.77.0 or later.
• aks-preview extension 19.0.0b15 or later.

# Install or update the aks-preview extension
az extension add --name aks-preview
az extension update --name aks-preview

Register the preview feature​

az feature register --name AKS-AutomaticHostedSystemProfilePreview --namespace Microsoft.ContainerService

Create the cluster​

Select a region where managed system node pools are available. Check the supported regions for managed system node pools.

Set your variables​

RESOURCE_GROUP="myResourceGroup"
CLUSTER_NAME="myAKSCluster"
LOCATION="westcentralus"  # Choose a supported region (see: https://aka.ms/aks/automatic/managed-systempool-regions)

Create the resource group​

az group create --name $RESOURCE_GROUP --location $LOCATION

Create an Automatic cluster with a managed system node pool​

az aks create \
--resource-group $RESOURCE_GROUP \
--name $CLUSTER_NAME \
--location $LOCATION \
--sku automatic \
--enable-hosted-system

The output includes "hostedSystemProfile": { "enabled": true } confirming the feature is active.

Connect to the cluster and deploy an application​

Get credentials for your cluster and deploy the AKS Store demo application:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

kubectl create ns aks-store-demo
kubectl apply -n aks-store-demo -f https://aka.ms/aks/quickstarts/store.yaml

Check the ingress address and open it in your browser once an IP is assigned:

kubectl get ingress store-front -n aks-store-demo --watch

Your workload runs on user node pools in your subscription that Node Auto Provisioning will create, while system services stay on the managed pool.

The managed system nodes will not be running in your Azure subscription.

Looking ahead​

Upcoming improvements include custom virtual network support, optimized platform components with reduced resource overhead, faster cluster provisioning, and a streamlined path to Deployment Safeguards compliance. Longer term, managed system node pools will extend to all existing AKS Automatic clusters.

Follow the AKS public roadmap for updates on these features.

Next steps​

Ready to get started?

1. Try it now: Follow the managed system node pools quickstart.
2. Share feedback: Open issues or ideas in AKS GitHub Issues.
3. Join the community: Subscribe to the AKS Community YouTube and follow @theakscommunity on X.

Share your experience with how managed system node pools simplify your operations and where the service can continue to improve.

Selecting a Linux OS option​

AKS has just released support for two new Linux OS options:

• Azure Linux OS Guard (preview) is Microsoft-created and optimized for Azure. OS Guard is built on top of Azure Linux with specialized configuration to support containerized workloads with security optimizations.
• Flatcar Container Linux for AKS (preview) is a CNCF-governed, vendor-neutral, container-optimized immutable OS, best suited for running on multi-cloud and on-prem environments.

As contributors to both projects, we understand the distinct customer needs each solution addresses. Customers running containerized workloads across multiple clouds and seeking consistency during critical OS updates typically choose Ubuntu or Flatcar Container Linux. Security-focused enterprises operating primarily on Azure and prioritizing a unified support experience opt for Azure Linux or OS Guard. Both approaches are valid, distinct, and fully supported by Microsoft.

What's different about optimized Linux OS options?​

The main optimization in OS options like Azure Linux OS Guard and Flatcar Container Linux is their immutability.

An immutable operating system refers to a type of operating system that cannot be modified at runtime. All OS binaries, libraries and static configuration are read-only, while the bit-for-bit integrity is often cryptographically protected. These special purpose operating systems usually come without any kind of package management or other traditional means of altering the OS, shipping as self-contained images. User workloads run in isolated environments like containers, sandboxed from the OS.

While these are certainly limiting factors compared to general purpose operating systems, immutable systems perform unparalleled in security and compliance:

• Binaries cannot be changed, eliminating whole classes of sandbox escapes and exploits.
• Special purpose operating systems include only what’s absolutely necessary, minimizing the attack surface.
• As individual parts of the OS cannot be swapped in or out, any given OS release always corresponds to the full version-set of all software and libraries included with that release. This significantly eases software inventory management and makes version drift impossible.

What’s more, immutable operating systems can bring similar benefits to node configuration. By applying node configuration at provisioning time only, there is no configuration drift. To phrase it differently, a node does not hold state, its state is defined by configuration passed during provisioning – making node provisioning reproducible.

While immutability is the core difference, there's typically more security features offered with Kubernetes optimized OS options:

Recommendations for Linux OS on AKS​

When deciding between which Linux OS options to use, we recommend the following:

• Use Flatcar Container Linux for AKS (preview) if you're looking for a vendor neutral, community stewarded immutable OS with cross-cloud support.
• Use Azure Linux OS Guard (preview) if you're looking for an immutable OS that is Microsoft-created and optimized for Azure.
• Use Ubuntu if you're looking for a portable, general purpose OS with cross-cloud support.
• Use Azure Linux if you're looking for a general purpose OS that is Microsoft-created and optimized for Azure.

Figure 1: Comparison across OS options supported on AKS, including Flatcar Container Linux for AKS, Azure Linux OS Guard, Ubuntu, and Azure Linux.

Migration to an optimized Linux OS option​

If you'd like to migrate to Azure Linux OS Guard (preview) or Flatcar Container Linux for AKS (preview), you'll want to keep in mind the following limitations and recommendations.

Immutable operating systems, by implication, make large parts of a node’s file system read-only. While Kubernetes workloads in general should not break abstraction and interfere with a node’s OS, the reality is often different. Care must be taken when migrating from general purpose operating systems. We have observed workloads’ expectations not being uniformly upheld on immutable systems particularly with, but not limited to:

• Any containers that require access to the host filesystem (e.g. via a /host/... mount), in particular init containers and daemonsets.
• Containers required to run in host PID and / or Networking namespace

Some AKS features may not be supported when using Azure Linux OS Guard (preview) or Flatcar Container Linux for AKS (preview). If you are using a feature that is not supported by the new OS, you will not be able to migrate your existing clusters/node pools. Ensure you review the limitations called out in their respective pages thoroughly.

When planning to migrate to an optimized OS option, we recommend the following:

• Ensure your workloads configure and run successfully on the new OS in test/dev before migrating any production clusters.
• If you'd like to migrate existing Linux clusters or node pools to Azure Linux OS Guard (preview), you can use in-place OS Sku migration. There are pre-requisites and limitations to this process, see documentation for details.
• If you'd like to migrate to Flatcar Container Linux for AKS (preview), you'll need to create new clusters and/or node pools and migrate existing workloads. Flatcar is available on all AKS supported Kubernetes versions.

Community Stewardship​

AKS is built on community stewarded open source projects. Our teams maintain Azure Linux OS Guard, contribute significantly to Flatcar Container Linux, and actively collaborate with the Immutable Linux community, the UAPI group, and other open source initiatives. Our continued engagement with projects like Flatcar improves the ecosystem for everybody and also empowers our users and customers to actively engage and participate in both development as well as project stewardship – driving the technology as well as determining the course and direction of these projects.

Join us in the Flatcar Container Linux open source project, which is community-driven and governed by the Cloud Native Computing Foundation. Get involved, contribute, and help shape the future of Flatcar Container Linux:

• Flatcar's participation how-to
• Chat with Flatcar contributors over at Matrix

Roadmap​

We’re excited to continue to extend AKS support for these optimized OS options. Our long-term goals include:

• In-place updates of OS and Kubernetes: faster, safer, less resource constraining
• Trusted and Confidential computing, locked-down execution through code signing
• Making signed execution available to everyone, by means of multiple trust levels and the option for users to use their own signing keys for their workloads

As we build these new features, we will be excited to share all these achievements with the broader Linux and Kubernetes ecosystem by contributing back and by making building blocks available.

To follow along with our backlog and progress, please see our public roadmaps:

• Flatcar Container Linux roadmap
• AKS Public Roadmap
• Azure Linux roadmap

Questions?​

Connect with the AKS and Azure Linux teams and communities through our GitHub discussions or share your feedback and suggestions.

We're announcing the preview availability of nftables mode for kube-proxy in Azure Kubernetes Service (AKS). This feature was requested in GitHub issue #5061 and is now aligned with the upstream Kubernetes GA release of the nftables backend.

Background​

Kubernetes 1.33 introduced nftables as a fully supported kube-proxy mode. It serves as the modern replacement for iptables, offering a more efficient rule model and improved performance characteristics on newer Linux kernels. As highlighted by the upstream project, nftables reduces rule churn and avoids the scaling and latency limitations seen in large clusters using iptables.

Unlike iptables, which implements the ruleset in an O(n) manner that slows down processing as the number of services grows, nftables utilizes a structure with a roughly O(1) map lookup. As a result, packet processing time is more or less constant regardless of cluster size, and the best/average/worst cases are very similar:

In clusters with 5,000 and 10,000 Services, the p50 (average) latency for nftables is approximately the same as the p01 (best-case) latency for iptables. In the 30,000 Service cluster, the p99 (worst-case) latency for nftables manages to beat the p01 latency for iptables by a few microseconds! Here's both sets of data together, though you may have to squint to see the nftables results:

For additional context, see the upstream GA announcement: Kubernetes blog: NFTables mode for kube-proxy.

What's available in AKS​

AKS now exposes nftables through the kube-proxy configuration preview feature. You can configure kube-proxy in one of three modes:

• IPTABLES
• IPVS
• NFTABLES (new – preview)

This configuration is applied through --kube-proxy-config during cluster creation or update.

Example:

{
  "enabled": true,
  "mode": "NFTABLES"
}

Enabling the preview​

1. Install the latest aks-preview CLI extension.
2. Register the KubeProxyConfigurationPreview feature flag.
3. Create or update your cluster with the nftables kube-proxy configuration.

Full details are in the updated documentation: Configure kube-proxy in AKS.

Notes for operators​

• Switching kube-proxy modes may cause brief disruptions as rules are reprogrammed.

Feedback​

We encourage you to try the feature in non-production clusters and share feedback in the original GitHub thread:
GitHub issue #5061

Running GPU workloads on AKS enables scalable, automated data processing and AI applications across Windows, Ubuntu, or Azure Linux nodes. Azure Linux, Microsoft’s minimal and secure OS, simplifies GPU setup with validated drivers and seamless integration, reducing operational efforts. This blog covers how AKS supports GPU nodes on various OS platforms and highlights the security and performance benefits of Azure Linux for GPU workloads.

Unique challenges of GPU nodes​

Deploying a GPU workload isn’t just about picking the right VM size. There is also significant operational overhead that developers and platform engineers need to manage.

We’ve found that many of our customers struggled to manage GPU device discoverability/scheduling and observability, especially across different OS images. Platform teams spent cycles maintaining custom node images and post-deployment scripts to ensure CUDA compatibility, while developers had to debug “GPU not found” errors or stalled workloads that consumed GPU capacity with limited visibility into utilization.

The inconsistent experience across OS options on AKS was a major challenge that we sought to improve. We wanted to encourage our customers to use the OS that best-fit their needs, not blocking them because of feature parity gaps.

For example, Azure Linux support for GPU-enabled VM sizes on AKS was historically limited to NVIDIA V100 and T4, creating a gap for Azure Linux customers requiring higher-performance options. Platform teams looking to run compute-intensive workloads, such as general-purpose AI/ML workloads or large-scale simulations, were unable to do so with Azure Linux and NVIDIA NC A100 GPU node pools -- until now.

AKS expanding Azure Linux GPU support​

NC A100 GPU support​

The introduction of Azure Linux 3.0 support for NC A100 GPU node pools in AKS starts to close many of these gaps. For platform engineers, the new OS image standardizes the underlying kernel, container runtime, and driver stack while enabling GPU provisioning in a single declarative step. Instead of layering custom extensions or maintaining golden images, engineers can now define a node pool with --os-sku AzureLinux and get a consistent, secure, and AKS-managed runtime that includes NVIDIA drivers/plugin setup and GPU telemetry out of the box. The Azure Linux 3.0 image also aligns with the AKS release cadence, which means fewer compatibility issues when upgrading clusters or deploying existing workloads onto GPU nodes.

AKS fully managed GPU nodes (preview)​

Using NVIDIA GPUs with Azure Linux on AKS requires the installation of several components for proper functioning of AKS-GPU enabled nodes, including GPU drivers, the NVIDIA Kubernetes device plugin, and GPU metrics exporter for telemetry. Previously, the installation of these components was either done manually or via the open-source NVIDIA GPU Operator, creating operational overhead for platform engineers. To ease this complexity and overhead, AKS has released support for fully managed GPU nodes (preview), which installs the NVIDIA GPU driver, device plugin, and Data Center GPU Manager (DCGM) metrics exporter by default.

Deploying GPU workloads on AKS with Azure Linux 3.0​

Customers choose to run their GPU workloads on Azure Linux for many reasons, such as the security posture, support model, resiliency, and/or performance optimizations that the OS provides. Some of the benefits that Azure Linux provides for your GPU workloads include:

Deploy a GPU workload with Azure Linux on AKS​

Deploying your GPU workloads on AKS with Azure Linux 3.0 is simple. Let’s use the newly supported NVIDIA NC A100 GPU as our example.

1. To add an NVIDIA NC A100 node pool running on Azure Linux to your AKS cluster using the fully managed GPU node experience you can follow these instructions. Please note, the following parameters must be specified in your az aks nodepool add command to create an NVIDIA NC A100 node pool running on Azure Linux:

   • --os-sku AzureLinux: provisions a node pool with the Azure Linux container host as the node OS.
   • --node-vm-size Standard_nc24ads_A100_v4: provisions a node pool using the Standard_nc24ads_A100_v4 VM size. Please note, any of the sizes in the Azure NC_A100_v4 series are supported.

2. With the DCGM exporter installed by default, you can observe detailed GPU metrics such as utilization, memory consumption, and error states.

If you prefer not to use a preview feature, you can follow these instructions on AKS to create an NVIDIA NC A100 node pool with Azure Linux by manually installing the NVIDIA device plugin via a DaemonSet. You’ll also need to manually install the DCGM exporter to consume GPU metrics.

Observability & monitoring​

Monitoring GPU performance is critical for optimizing utilization, troubleshooting workloads, and enabling cost-efficient scaling in AKS clusters. Traditionally, NVIDIA GPU node pools were treated as opaque resources - jobs would succeed or fail without visibility into whether GPUs were fully utilized or misallocated.

With the DCGM exporter now managed on AKS, cluster operators can collect detailed GPU metrics such as utilization, memory consumption, and error states for analysis. These metrics can integrate naturally with existing observability pipelines, providing a foundation for intelligent automation and alerting.

As an example, a platform team can configure scaling logic in the Cluster Autoscaler (CAS) or Kubernetes Event-Driven Autoscaling (KEDA) to add A100 nodes when GPU utilization exceeds 70%, or scale down when utilization remains low for a defined interval. This enables GPU infrastructure to operate as a dynamic, demand-driven resource rather than a static, high-cost allocation.

For more conceptual guidance on GPU metrics in AKS, visit these docs.

What's next?​

The Azure Linux and AKS teams are actively working on expanding support for additional GPU VM sizes and managed GPU features on AKS. You can expect to see Azure Linux support for the NVIDIA ND A100, NC H100, and ND H200 families landing in the near future, as well as Azure Linux support for managed AKS GPU features like multi-instance GPU (MIG), built-in GPU metrics in Azure Managed Prometheus and Grafana, and KAITO.

Using GPUs in Kubernetes​

Today, you need a few vendor-specific software components to even get started with a GPU node pool on Kubernetes - in particular, these are the Kubernetes device plugin and the GPU drivers. While the installation of GPU drivers makes sense, you might ask - why do we need a specific device plugin to be installed as well?

Since Kubernetes does not have native support for special devices like GPUs, the device plugin surfaces and makes these resources available to your application. The device plugin works by exposing the number of GPUs on a node, by taking in a list of allocatable resources through the device plugin API and passing this to the kubelet. The kubelet then tracks this set and inputs the count of arbitrary resource type on the node to API Server, for kube-scheduler to use in pod scheduling decisions.

However, there are some limitations with this device plugin approach. It only allows GPUs to be statically assigned to a workload, without any ability for fine-grained sharing, partitioning, nor hot-swapping/reconfiguring of the GPUs.

The introduction of DRA provides a path to addressing some of these limitations, by ensuring resources can by dynamically categorized, requested, and used in a cluster. The Dynamic Resource Allocation API generalizes the Persistent Volumes API for generic resources, like GPUs. It allows for resource adjustment based on real-time demand and proper configuration without manual intervention.

NVIDIA's DRA driver extends this capability for their GPUs, introducing GPUs and ComputeDomains as two types of resources users can manage through DRA handles.

Let’s backtrack with a simpler scenario… I’m confused​

Imagine that Kubernetes DRA is a working kitchen, and its staff is preparing a multi-course meal. In this example, the ingredients include:

• Pods: The individual dishes that need to be prepared
• Nodes: Various kitchen stations where the dishes are cooked
• API Server: The head chef, directing the entire operation
• Scheduler: The sous-chef, making sure each dish is cooked at the right station
• Resource Quotas: Portion sizes, ensuring each dish uses a specific amount of ingredients
• Horizontal Pod Autoscaler (HPA): Dynamic serving staff, adjusting the number of dishes served based on customer demand

The instructions for this DRA recipe are then to:

1. Prep: Start by defining the recipes (Pods) and gathering all ingredients (resources like CPU and memory). Each recipe is assigned to a station (Node) by the head chef (API Server) through the sous-chef (Scheduler).
2. Cooking: The stations (Nodes) start cooking the dishes (Pods) as directed. The head chef ensures everything is progressing smoothly and the dishes are being prepared according to the recipes.
3. Portion Control: Use Resource Quotas to manage the portion sizes of each dish. This ensures that no single dish uses up too many ingredients, maintaining a balanced kitchen.
4. Dynamic Serving: The HPA acts like a waiter, monitoring the number of customers (resource usage metrics) and adjusting the number of dishes prepared (Pod count) dynamically. If more customers arrive, more dishes are prepared; if the demand decreases, fewer dishes are made.
5. Serving the Meal: The kitchen operates efficiently, with each dish receiving the right amount of attention and resources needed to ensure a delightful dining experience for the customers.

Just like in a restaurant kitchen, Kubernetes DRA makes sure your applications (dishes) get the right number of resources (ingredients) at the right time, providing an efficient resource management experience in the complex environment.

Now, we can dig into the nitty-gritty​

DRA involves several key components to efficiently manage resources within a cluster, shown in the following sample resource driver:

apiVersion: resource.k8s.io/v1alpha3
kind: DeviceClass
name: resource.example.com
spec:
  selectors:
  - cel:
      expression: device.driver == "resource-driver.example.com"
---
apiVersion: resource.k8s.io/v1alpha1
kind: ResourceClaimTemplate
metadata:
  name: gpu-template
spec:
  spec:
    resourceClassName: resource.example.com

–--
apiVersion: v1
kind: Pod
metadata:
  name: pod
spec:
  containers:
  - name: container0
    image: gpuDriverImage
    command: ["cmd0"]
    resources:
      claims:
      - name: gpu
  - name: container1
    image: gpuDriverImage
    command: ["cmd0"]
    resources:
      claims:
      - name: gpu
  resourceClaims:
  - name: gpu
    source:
      resourceClaimTemplate: gpu-template

Adapted from: DRA for GPUs in Kubernetes

Here's what each of the components are:

The example above demonstrates GPU sharing within a pod, where two containers get access to one ResourceClaim object created for the pod.

The Kubernetes scheduler will try multiple times to identify the right node and let the DRA resource driver(s) know that the node is ready for resource allocation. Then the resourceClaim stages record that the resource has been allocated on a particular node, and the scheduler is signaled to run the pod. (This prevents pods from being scheduled onto “unprepared” nodes that cannot accept new resources.) Leveraging the Named Resources model, the DRA resource driver can specify:

1. How to get a hold of a specific GPU type, and
2. How to configure the GPU that has been allocated to a pod and assigned to a node.

In the place of arbitrary resource count, an entire object now represents the choice of resource. This object is passed to the scheduler at node start time and may stream updates if resources become unhealthy.

Pulling this all together, the key components of DRA look like:

Vendor specific drivers​

Vendors can provide driver packages that extend the base DRA capabilities to interact with their own resources. We will take a look at NVIDIA's DRA drivers, and explore how that allows for flexible and dynamic allocation of their GPUs.

Seeing NVIDIA's DRA driver in action​

Now, let’s see how NVIDIA's open-source k8s DRA driver works on a Kubernetes cluster. We'll walk through:

1. Setting up your cluster and DRA drivers
2. Verifying your drivers are installed properly
3. Running a sample workload to illustrate a GPU being flexibly assigned

Before you begin​

• If you don't have a cluster, create one using the Azure CLI, Azure PowerShell, Azure portal, or IaaC tool of your choice. Here's an example of creating one using the Azure CLI. Note that the cluster should be on Kubernetes v.134 or later to have the DRA feature gate enabled.

  az aks create --name myAKSCluster --resource-group myResourceGroup --location <region>  --kubernetes-version 1.34
  

• Your GPU node pool should be provisioned with an NVIDIA GPU enabled VM size.

  • Make sure you also skip GPU driver installation, as we install the drivers via the NVIDIA GPU operator in this tutorial.

  az aks nodepool add --cluster-name myAKSCluster --resource-group myResourceGroup --name gpunodepool  --node-count 1 --gpu-driver none --node-vm-size Standard_NC6s_v3 (or alternative NVIDIA GPU SKU)
  

Get the credentials for your cluster​

Get the credentials for your AKS cluster using the az aks get-credentials command. The following example command gets the credentials for the myAKSCluster in the myResourceGroup resource group:

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster

Verify DRA is enabled​

You can confirm whether or not DRA is enabled on your cluster by checking deviceclasses and resourceslices.

Check deviceclasses via kubectl get deviceclasses or check resourceslices via kubectl get resourceslices.

The results for both should look similar to:

No resources found

If DRA isn't enabled on your cluster, you may instead get a response similar to error: the server doesn't have a resource type "deviceclasses"/"resourceslices".

Install the NVIDIA GPU operator​

Set up your GPU operator, ensure GPUs are schedulable, and GPU workloads can be run successfully.

[!NOTE] Make sure you use a version of the GPU operator that matches or exceeds the version you specify when installing the DRA driver.

1. Install the GPU operator. We will be using the DRA drivers to manage our GPUs, so we also want to ensure we disable the Kubernetes device plugin during install. We will use an operator-install.yaml to parameters we'd like the operator to be installed with.

   • Create operator-install.yaml like so:

     devicePlugin:
      enabled: false
     driver:
        enabled: true
      toolkit:
        env:
          # Limits containers running in _unprivileged_ mode from requesting access to arbitrary GPU devices 
          - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
            value: "false"
     

   • Install the GPU operator

     helm install --wait --generate-name -n gpu-operator \
     --create-namespace nvidia/gpu-operator \
     --version=v25.10.0 \
     -f operator-install.yaml
     

2. Make sure all your GPU operator components are running and ready via kubectl get pod -n gpu-operator. The result should look similar to:

   NAME                                                              READY   STATUS      RESTARTS   AGE
   gpu-feature-discovery-t9xs5                                       1/1     Running     0          2m9s
   gpu-operator-1761843468-node-feature-discovery-gc-6648dd8449tbx   1/1     Running     0          2m27s
   gpu-operator-1761843468-node-feature-discovery-master-597bhvwmm   1/1     Running     0          2m27s
   gpu-operator-1761843468-node-feature-discovery-worker-mvbbt       1/1     Running     0          2m27s
   gpu-operator-f8577988-p2k9x                                       1/1     Running     0          2m27s
   nvidia-driver-daemonset-tgf78                                     1/1     Running     0          1m30s
   nvidia-container-toolkit-daemonset-sqchb                          1/1     Running     0          2m10s
   nvidia-cuda-validator-f7g97                                       0/1     Completed   0          77s
   nvidia-dcgm-exporter-6lbxc                                        1/1     Running     0          2m9s
   nvidia-device-plugin-daemonset-v74ww                              1/1     Running     0          2m9s
   nvidia-mig-manager-jsnkr                                          1/1     Running     0          16s
   nvidia-operator-validator-h8s5n                                   1/1     Running     0          2m10s
   

Installing NVIDIA DRA drivers​

The recommended way to install the driver is via Helm. Ensure you have Helm updated to the correct version.

1. Add the Helm chart that contains the DRA driver.

   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
   

2. Create a dra-install.yaml to specify parameters during the installation.

    gpuResourcesEnabledOverride: true
    resources-computeDomains:
      enabled: false # We'll be using GPUs, not compute domains.
    controller:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.azure.com/mode
                operator: In
                values:
                - system   # Makes sure system nodes are utilized 
    nvidiaDriverRoot: "/run/nvidia/driver"
   

   1. Install the DRA driver.

   helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
          # Ensure you select an appropriate version (https://github.com/NVIDIA/k8s-dra-driver-gpu/releases)
          --version="25.8.0" \
          --create-namespace \
          --namespace nvidia-dra-driver-gpu \
          -f dra-install.yaml \
   

Verify your installation​

Once setup, double check whether all DRA driver components are ready and running:

$ kubectl get pod -n nvidia-dra-driver-gpu
NAME                                               READY   STATUS    RESTARTS   AGE
nvidia-dra-driver-gpu-kubelet-plugin-[...]         1/1     Running   0          61m

deviceclasses and resourceslices should also recognize the new GPU devices. You can use kubectl get deviceclasses or kubectl get resourceslices to confirm.

Run a GPU workload using DRA drivers​

You can run some sample workloads to confirm that DRA drivers are installed and behave as expected.

1. Create a namespace that houses the resources for our sample workloads.

   kubectl create namespace dra-gpu-share-test
   

2. Create a new ResourceClaimTemplate that is used to create ResourceClaims of 1 GPU for associated workloads. Save this manifest as my-rct.yaml.

   apiVersion: resource.k8s.io/v1
   kind: ResourceClaimTemplate
   metadata:
     namespace: dra-gpu-share-test
     name: single-gpu
   spec:
     spec:
       devices:
         requests:
         - name: gpu
           exactly:
             count: 1
             deviceClassName: gpu.nvidia.com
   

3. Create a pod manifest, dra-rct-pod.yaml, that takes advantage of our ResourceClaimTemplate. We spin up a pod that holds two containers, ctr0 and ctr1. Both containers reference the same ResourceClaim and therefore share access to the same GPU device.

   apiVersion: v1
   kind: Pod
   metadata:
     namespace: dra-gpu-share-test
     name: pod
     labels:
       app: pod
   spec:
     containers:
     - name: ctr0
       image: ubuntu:22.04
       command: ["bash", "-c"]
       args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
       resources:
         claims:
         - name: shared-gpu
     - name: ctr1
       image: ubuntu:22.04
       command: ["bash", "-c"]
       args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
       resources:
         claims:
         - name: shared-gpu
     resourceClaims:
     - name: shared-gpu
       resourceClaimTemplateName: single-gpu
     tolerations:
     - key: "nvidia.com/gpu"
       operator: "Exists"
       effect: "NoSchedule"
   

4. Fetch the containers' logs to check the GPU UUID for both containers

   kubectl logs pod -n dra-gpu-share-test --all-containers --prefix
   

5. The output should look similar to:

   [pod/pod/ctr0] GPU 0: NVIDIA H100 NVL (UUID: GPU-c552c7e1-3d44-482e-aaaf-507944ab75f7)
   [pod/pod/ctr1] GPU 0: NVIDIA H100 NVL (UUID: GPU-c552c7e1-3d44-482e-aaaf-507944ab75f7)
   

The results show us that the GPU UUID for both containers matches, confirming that both containers are accessing the same GPU device.

Next steps​

• Further validate your installation of the DRA drivers with sample workloads
• Learn more about NVIDIA DRA drivers

Questions?​

Connect with the AKS team through our GitHub discussions or share your feedback and suggestions.

No immediate action required​

Microsoft understands that customers value continuity and clarity around the maintenance and evolution of the components that power their workloads. There is no change or immediate action required today for AKS clusters using the Application Routing add-on with NGINX to manage Ingress NGINX resources. Microsoft will provide official support for Application Routing add-on Ingress NGINX resources through November 2026 and only for critical security patches during this period.

We are actively investing in the future of application connectivity in Azure Kubernetes Service (AKS), centered on the Gateway API. This includes support for the Gateway API in the Istio-based service mesh add-on, expanding the Application Routing add-on to support the Gateway API, and continuing investment in Application Gateway for Containers.

The future of the Application Routing add-on​

Application Routing with Gateway API, which will be powered by the Istio control plane for Gateway API-based ingress only, is planned for the first half of 2026, along with migration guidance documentation. The Kubernetes Gateway API represents the next generation of Kubernetes ingress traffic management, evolving from the Ingress API by offering richer routing capabilities, standardized extensibility, and a more secure, role-oriented design.

Alternative migration paths​

Alternatively, existing users of Ingress NGINX, including users who provisioned it through Application Routing add-on, can also migrate to one of the following options:

• The Istio-based service mesh add-on using the Gateway API
• Application Gateway for Containers using either the Ingress API or the Gateway API

For further questions, you can reach us on GitHub or create a support case.

CloudNativePG now anchors our recommended pattern for running production-ready PostgreSQL on AKS. With the latest feedback from EnterpriseDB (EDB), the documentation aligns to the refreshed container image catalogs, safer operator rollouts, and updates to the Barman backup tool that help teams meet availability targets from day one.

The updated set of articles walks through the journey end-to-end: an overview of the architecture, infrastructure setup with workload identity and storage RBAC, deployment with the new PostgreSQL 18 image, and day-two validation. You'll see how we folded in the CNPG controller tuning, the move toward self-managed PodMonitors, and guidance on planning for the Barman Cloud plugin as upstream support evolves.

We also expanded the operational coverage: monitoring with Prometheus and Grafana, exercising failover across availability zones, and validating backup and restore flows that rely on AKS workload identity. The how-tos keep the commands concise so you can get into the CLI quickly without losing sight of the bigger architectural decisions.

CloudNativePG on AKS, with EDB​

For production support for CloudNativePG on AKS, visit EDB's Azure Marketplace offering. As maintainers of the operator, they provide services to keep clusters compliant, optimized, and ready for future releases.

Thanks again to the CloudNativePG team at EDB for their collaboration, reviews, and continued support as we bring the latest PostgreSQL best practices to AKS users. Dive in and see how quickly you can stand up a resilient PostgreSQL footprint on AKS.

By default, AKS and Azure Monitor give you a rich set of out-of-the-box insights: CPU and memory utilization, pod restarts, node health, and Kubernetes control plane metrics. But many teams need more visibility into what’s happening inside their workloads — for example:

• Application-specific metrics such as API request latency or queue depth
• Custom business metrics like transactions per second or user sessions
• System-level data such as network interface stats, disk I/O, or custom log counters

Traditionally, enabling this kind of deep observability required deploying and managing a full Prometheus stack — configuring storage, scaling scrapers, and handling upgrades. That adds operational complexity, especially when all you need is a few targeted custom metrics. This is where Azure Monitor managed service for Prometheus comes in — it takes care of high availability, storage, and scaling, so you can focus entirely on defining the metrics that matter most. And by using Telegraf as a lightweight collector, you can easily publish custom metrics from your workloads or nodes directly into your managed monitoring environment, with no self-managed Prometheus servers required.

While our example uses network metrics, the same pattern applies to any custom data source you want to monitor in AKS. If you want to take this example one step further, we have a hands-on experience with the AKS Labs: Advanced Observability Concepts and the Observability with Managed Prometheus and Managed Grafana at the Microsoft Reactor.

A common question we hear from Kubernetes users is:

"How can I scrape a specific set of custom metrics from my cluster without adding too much operational overhead?"

Before we dive into the setup, let’s look at why this approach is so effective for extending AKS monitoring.

Why This Matters​

If you run workloads on AKS, you already get a solid baseline of metrics out of the box — node and pod resource usage, cluster health, and control plane telemetry through Azure Monitor. But those built-in signals don’t always tell the full story.

Engineers often need visibility into what’s actually happening inside their workloads or on the host — things like:

• Network throughput or packet drops on specific interfaces.
• Application-level metrics like queue depth or request latency.
• Custom counters from scripts, logs, or local daemons.

You could deploy a full Prometheus stack to get those insights, but that means managing storage, scaling scrapers, maintaining alert rules, and patching over time. For many teams, that’s more operational effort than it’s worth — especially when you just need a handful of custom metrics.

This approach combines Telegraf, Azure Monitor managed service for Prometheus, and Azure Managed Grafana to bridge that gap. Telegraf runs as a DaemonSet, collecting metrics from every node (or from any command or script you define) and exposing them in Prometheus format. Azure Monitor managed service for Prometheus then handles a) scraping, b) scaling, and c) storage — so there’s no local Prometheus to manage — and Grafana provides dashboards and alerting without extra infrastructure.

The result is a lightweight, fully managed way to extend AKS observability with exactly the metrics you care about, using standard open-source tools and Azure’s managed services.

Solution at a Glance​

Here’s the workflow we’re setting up:

+-----------------+    +------------------+    +-----------------+ 
|   AKS Nodes     |    |  Azure Managed   |    |  Azure Managed  | 
|                 |    |   Prometheus     |    |    Grafana      | 
| +-------------+ |    |                  |    |                 | 
| |  Telegraf   | |--->|  Scrapes via     |--->|  Dashboards &   | 
| | DaemonSet   | |    |  PodMonitor      |    |  Alerting       | 
| |:2112/metrics| |    |                  |    |                 | 
| +-------------+ |    +------------------+    +-----------------+ 
+-----------------+                                                

• Telegraf DaemonSet: runs on every node and collects your custom metrics.
• Prometheus PodMonitor: automatically scrapes those metrics endpoints.
• Azure Managed Grafana: visualizes and alerts on metrics without extra servers.

Understanding the solution​

For our example, we will create a custom collection of the following metrics for each network interface using ip -s link:

Each metric includes the following labels:

• cluster: AKS cluster identifier
• environment: Environment tag (configurable)
• host: Node hostname
• hostname: Node hostname (duplicate for compatibility)
• interface: Network interface name (eth0, eth1, etc.)
• state: Interface operational state

Setup your environment variables and placeholders​

In these next steps, we will set up a new AKS cluster, an Azure Managed Grafana instance, and an Azure Monitor Workspace.

export RG_NAME="rg-telegraf-on-aks"
export LOCATION="westus3"

# Azure Kubernetes Service Cluster
export AKS_CLUSTER_NAME="telegraf-on-aks"

# Azure Managed Grafana
export GRAFANA_NAME="aks-blog-${RANDOM}"

# Azure Monitor Workspace
export AZ_MONITOR_WORKSPACE_NAME="telegraf-on-aks"

Next, let's create our solution:

# Create resource group
az group create --name ${RG_NAME} --location ${LOCATION}

# Create an Azure Monitor Workspace
az monitor account create \
  --resource-group ${RG_NAME} \
  --location ${LOCATION} \
  --name ${AZ_MONITOR_WORKSPACE_NAME}

# Get the Azure Monitor Workspace ID
AZ_MONITOR_WORKSPACE_ID=$(az monitor account show \
  --resource-group ${RG_NAME} \
  --name ${AZ_MONITOR_WORKSPACE_NAME} \
  --query id -o tsv)

Create a Grafana instance. The Azure CLI extension for Azure Managed Grafana (amg) will be used for this.

# Add the Azure Managed Grafana extension to az cli:
az extension add --name amg

# Create an Azure Managed Grafana instance:
az grafana create \
  --name ${GRAFANA_NAME} \
  --resource-group $RG_NAME \
  --location $LOCATION

# Once created, save the Grafana resource ID
GRAFANA_RESOURCE_ID=$(az grafana show \
  --name ${GRAFANA_NAME} \
  --resource-group ${RG_NAME} \
  --query id -o tsv)

We can now create the cluster, passing both the 'grafana-resource-id' and 'azure-monitor-workspace-resource-id' during cluster creation:

# Create the AKS cluster
az aks create \
  --name ${AKS_CLUSTER_NAME}  \
  --resource-group ${RG_NAME} \
  --node-count 1 \
  --enable-managed-identity  \
  --enable-azure-monitor-metrics \
  --grafana-resource-id ${GRAFANA_RESOURCE_ID} \
  --azure-monitor-workspace-resource-id ${AZ_MONITOR_WORKSPACE_ID}

# Get the cluster credentials
az aks get-credentials \
  --name ${AKS_CLUSTER_NAME} \
  --resource-group ${RG_NAME}

Verify that the PodMonitor CRD is now available in your cluster

# Check if PodMonitor CRD exists
kubectl get crd | grep podmonitor

# Expected output (Azure Monitor managed service for Prometheus):
# podmonitors.azmonitoring.coreos.com                  2025-07-23T19:12:02Z

Deploying the solution​

We’ll deploy a single YAML manifest that contains:

• ConfigMap for Telegraf config + your custom metric script (parse_ip_stats.sh)
• DaemonSet to run Telegraf on each node
• ServiceAccount, Service, and PodMonitor

1. Create the Telegraf Configuration

   First, let's create the main Telegraf configuration:

   cat <<EOF > 01-configmap.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
   name: telegraf-config
   namespace: default
   data:
   telegraf.conf: |
      [global_tags]
         environment = "aks"
         cluster = "aks-telegraf"
   
      [agent]
         interval = "30s"
         round_interval = true
         metric_batch_size = 1000
         metric_buffer_limit = 10000
         collection_jitter = "5s"
         flush_interval = "30s"
         flush_jitter = "5s"
         precision = ""
         hostname = "\$HOSTNAME"
         omit_hostname = false
   
      # Custom script to parse ip -s link output
      [[inputs.exec]]
         commands = ["/usr/local/bin/parse_ip_stats.sh"]
         timeout = "10s"
         data_format = "influx"
         name_override = "network_interface_stats"
   
      # Prometheus metrics output
      [[outputs.prometheus_client]]
         listen = ":2112"
         metric_version = 2
         path = "/metrics"
         expiration_interval = "60s"
         collectors_exclude = ["gocollector", "process"]
   EOF
   

2. Create the Network Parsing Script

   Now create the ConfigMap containing our custom script that parses network interface statistics:

   cat <<EOF > 02-scripts-configmap.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
   name: telegraf-scripts
   namespace: default
   data:
   parse_ip_stats.sh: |
      #!/bin/bash
      # Script to parse ip -s link output and convert to InfluxDB line protocol
      
      # Get the current timestamp in nanoseconds
      timestamp=\$(date +%s%N)
      hostname=\$(hostname)
      
      # Parse ip -s link output for network statistics
      ip -s link | awk -v ts="\$timestamp" -v host="\$hostname" '
      BEGIN {
         interface = "";
         state = "";
         mtu = 0;
      }
      
      # Parse interface line (e.g., "2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...")
      /^[0-9]+:/ {
         # Extract interface name (handle both regular and @ notation)
         if (match(\$0, /^[0-9]+: ([^:@]+)/)) {
               interface_match = substr(\$0, RSTART, RLENGTH);
               # Remove the number and colon prefix, then trim spaces
               gsub(/^[0-9]+: */, "", interface_match);
               interface = interface_match;
         }
         
         # Extract state from flags
         if (match(\$0, /<[^>]+>/)) {
               flags = substr(\$0, RSTART+1, RLENGTH-2);
               if (index(flags, "UP")) {
                  state = "up";
               } else {
                  state = "down";
               }
         }
         
         # Extract MTU
         if (match(\$0, /mtu [0-9]+/)) {
               mtu_str = substr(\$0, RSTART+4, RLENGTH-4);
               mtu = mtu_str + 0;
         }
      }
      
      # Parse RX line header (RX: bytes packets errors dropped missed mcast)
      /^[[:space:]]*RX:.*bytes.*packets.*errors.*dropped.*missed.*mcast/ {
         getline; # Get the next line with the actual numbers
         gsub(/^[[:space:]]+/, ""); # Remove leading spaces
         n = split(\$0, rx_fields);
         if (n >= 6) {
               rx_bytes = rx_fields[1];
               rx_packets = rx_fields[2];
               rx_errors = rx_fields[3];
               rx_dropped = rx_fields[4];
               rx_missed = rx_fields[5];
               rx_multicast = rx_fields[6];
         }
      }
      
      # Parse TX line header (TX: bytes packets errors dropped carrier collsns)
      /^[[:space:]]*TX:.*bytes.*packets.*errors.*dropped.*carrier.*collsns/ {
         getline; # Get the next line with the actual numbers
         gsub(/^[[:space:]]+/, ""); # Remove leading spaces
         n = split(\$0, tx_fields);
         if (n >= 6 && interface != "" && interface != "lo") {
               tx_bytes = tx_fields[1];
               tx_packets = tx_fields[2];
               tx_errors = tx_fields[3];
               tx_dropped = tx_fields[4];
               tx_carrier = tx_fields[5];
               tx_collisions = tx_fields[6];
               
               # Output metrics after processing both RX and TX (skip loopback)
               printf "network_interface_stats,interface=%s,hostname=%s,state=\"%s\" ", interface, host, state;
               printf "mtu=%si,", mtu;
               printf "rx_bytes=%si,rx_packets=%si,rx_errors=%si,rx_dropped=%si,rx_missed=%si,rx_multicast=%si,", rx_bytes, rx_packets, rx_errors, rx_dropped, rx_missed, rx_multicast;
               printf "tx_bytes=%si,tx_packets=%si,tx_errors=%si,tx_dropped=%si,tx_carrier=%si,tx_collisions=%si ", tx_bytes, tx_packets, tx_errors, tx_dropped, tx_carrier, tx_collisions;
               printf "%s\n", ts;
         }
      }
      '
   EOF
   

3. Create the Service Account

   Create a service account for RBAC permissions:

   cat <<EOF > 03-serviceaccount.yaml
   apiVersion: v1
   kind: ServiceAccount
   metadata:
   name: telegraf-sa
   namespace: default
   EOF
   

4. Create the DaemonSet

   Now create the main DaemonSet that runs Telegraf on each node:

   cat <<EOF > 04-daemonset.yaml
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
   name: telegraf
   namespace: default
   labels:
      app: telegraf
   spec:
   selector:
      matchLabels:
         app: telegraf
   template:
      metadata:
         labels:
         app: telegraf
      spec:
         serviceAccountName: telegraf-sa
         hostNetwork: true
         hostPID: true
         tolerations:
         - key: node-role.kubernetes.io/master
         operator: Exists
         effect: NoSchedule
         - key: node-role.kubernetes.io/control-plane
         operator: Exists
         effect: NoSchedule
         containers:
         - name: telegraf
         image: telegraf:1.28
         env:
         - name: HOSTNAME
            valueFrom:
               fieldRef:
               fieldPath: spec.nodeName
         ports:
         - name: prometheus
            containerPort: 2112
            protocol: TCP
         securityContext:
            privileged: true
            runAsUser: 0
         volumeMounts:
         - name: telegraf-config
            mountPath: /etc/telegraf
         - name: telegraf-scripts
            mountPath: /scripts
         - name: proc
            mountPath: /host/proc
            readOnly: true
         - name: sys
            mountPath: /host/sys
            readOnly: true
         - name: var-run-docker
            mountPath: /var/run/docker.sock
            readOnly: true
         resources:
            requests:
               memory: "64Mi"
               cpu: "100m"
            limits:
               memory: "128Mi"
               cpu: "200m"
         command:
         - /bin/bash
         - -c
         - |
            # Install iproute2 if not present
            if ! command -v ip > /dev/null 2>&1; then
               apt-get update && apt-get install -y iproute2
            fi
            
            # Copy the parsing script to the expected location
            cp /scripts/parse_ip_stats.sh /usr/local/bin/parse_ip_stats.sh
            chmod +x /usr/local/bin/parse_ip_stats.sh
            
            # Start telegraf
            exec telegraf --config /etc/telegraf/telegraf.conf
         volumes:
         - name: telegraf-config
         configMap:
            name: telegraf-config
            defaultMode: 0755
         - name: telegraf-scripts
         configMap:
            name: telegraf-scripts
            defaultMode: 0755
         - name: proc
         hostPath:
            path: /proc
         - name: sys
         hostPath:
            path: /sys
         - name: var-run-docker
         hostPath:
            path: /var/run/docker.sock
         terminationGracePeriodSeconds: 30
   EOF
   

5. Create the Service

   Create a service to expose the Prometheus metrics endpoint:

   cat <<EOF > 05-service.yaml
   apiVersion: v1
   kind: Service
   metadata:
   name: telegraf-metrics
   namespace: default
   labels:
      app: telegraf
   spec:
   selector:
      app: telegraf
   ports:
   - name: prometheus
      port: 2112
      targetPort: 2112
      protocol: TCP
   type: ClusterIP
   EOF
   

6. Create the PodMonitor

   Finally, create the PodMonitor that tells Azure Monitor managed service for Prometheus to scrape our metrics:

   cat <<EOF > 06-podmonitor.yaml
   apiVersion: azmonitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
   name: telegraf-podmonitor
   namespace: default
   labels:
      app: telegraf
   spec:
   selector:
      matchLabels:
         app: telegraf
   podMetricsEndpoints:
   - port: prometheus
      interval: 30s
      path: /metrics
   EOF
   

7. Deploy All Components

   Now deploy all the components in order:

   kubectl apply -f 01-configmap.yaml
   kubectl apply -f 02-scripts-configmap.yaml
   kubectl apply -f 03-serviceaccount.yaml
   kubectl apply -f 04-daemonset.yaml
   kubectl apply -f 05-service.yaml
   kubectl apply -f 06-podmonitor.yaml
   

8. Verification

   After a minute or two, verify everything is running:

   kubectl get daemonset telegraf
   kubectl get pods -l app=telegraf
   kubectl get service telegraf-metrics
   kubectl get podmonitor telegraf-podmonitor
   

9. Validate the Metrics

   You can check if the new metrics are now being collected correctly, by forwarding the telegraf-metrics service port locally and then by running curl against it:

   kubectl port-forward svc/telegraf-metrics 2112:2112 &
   curl http://localhost:2112/metrics | head -20
   

   Sample output:

   # HELP network_interface_stats_rx_bytes Telegraf collected metric
   # TYPE network_interface_stats_rx_bytes untyped
   network_interface_stats_rx_bytes{interface="eth0",host="aks-node-1"} 16876971289
   

   Great! At this point we know that our collection is working. Next we will look into how to visualize these new metrics in Grafana.

Visualize in Grafana​

You can now go to your new Azure Managed Grafana instance and try some queries. To get the URL for your Azure Managed Grafana, run the following command:

GRAFANA_UI=$(az grafana show \
  --name ${GRAFANA_NAME} \
  --resource-group ${RG_NAME} \
  --query "properties.endpoint" -o tsv)

echo "Your Azure Managed Grafana is accessible at: $GRAFANA_UI"

Now that you know the URL, open Azure Managed Grafana and go to the Drilldown tab.

Make sure the Data Source is Managed_Prometheus_telegraf-on-aks.

Try to search for network_interface_ metrics. You should see all of the new metrics that are being collected by Telegraf

Next you can create table panels with Instant queries for top-N views or time series panels for trends over time. Here are some suggestions on metrics:

# Network throughput by node
sum(rate(network_interface_stats_rx_bytes[5m])) by (host)

# Top interfaces by traffic
topk(10, network_interface_stats_tx_bytes)

# Packet drops
sum(rate(network_interface_stats_rx_dropped[5m])) by (interface)

Cleaning up​

To remove these resources, you can run this command:

az group delete --name ${RG_NAME} --yes --no-wait

Conclusion​

In this post, we saw an approach to integrating custom metrics into Azure’s managed monitoring stack with minimal setup using Telegraf DaemonSet, for flexible metric collection, Azure Monitor managed service for Prometheus, for scraping and storage, and Azure Managed Grafana for visualization and alerting.

While our example used network metrics, the same pattern applies to any custom data source you want to monitor in AKS. If you want to take this example one step further, we have a hands-on experience with the AKS Labs: Advanced Observability Concepts and the Observability with Managed Prometheus and Managed Grafana at the Microsoft Reactor.

Modern language models now routinely exceed the compute and memory capacity of a single GPU or even a whole node with multiple GPUs on Kubernetes. Consequently, inference at the scale of billions of model parameters demands multi-node, distributed deployment. Frameworks like the open-source NVIDIA Dynamo platform play a crucial role by coordinating execution across nodes, managing memory resources efficiently, and accelerating data transfers between GPUs to keep latency low.

However, software alone cannot solve these challenges. The underlying hardware must also support this level of scale and throughput. Rack-scale systems like Azure ND GB200-v6 VMs, accelerated by NVIDIA GB200 NVL72, meet this need by integrating 72 NVIDIA Blackwell GPUs in a distributed GPU setup connected via high-bandwidth, low-latency interconnect. This architecture uses the rack as a unified compute engine and enables fast, efficient communication and scaling that traditional multi-node setups struggle to achieve.

For some more demanding or unpredictable workloads, even combining advanced hardware and distributed inference frameworks is not sufficient on its own. Inference traffic spikes unpredictably. Fixed, static inference configurations and setups with predetermined resource allocation can lead to GPU underutilization or overprovisioning. Instead, inference infrastructure must dynamically adjust in real time, scaling resources up or down to align with current demand without wasting GPU capacity or risking performance degradation.

A holistic solution: ND GB200-v6 VMs and Dynamo on AKS​

To effectively address the variability in inference traffic in distributed deployments, our approach combines three key components: ND GB200-v6 VMs, the NVIDIA Dynamo inference framework, with an Azure Kubernetes Service (AKS) cluster. Together, these technologies provide the scale, flexibility, and responsiveness necessary to meet the demands of modern, large-scale inference workloads.

ND GB200-v6: Rack-Scale Accelerated Hardware​

At the core of Azure’s ND GB200-v6 VM series is the liquid-cooled NVIDIA GB200 NVL72 system, a rack-scale architecture that integrates 72 NVIDIA Blackwell GPUs and 36 NVIDIA Grace™ CPUs into a single, tightly coupled domain.

The rack-scale design of ND GB200-v6 unlocks model serving patterns that were previously infeasible due to interconnect and memory bandwidth constraints.

NVIDIA Dynamo: a distributed inference framework​

NVIDIA Dynamo is an open source distributed inference serving framework that supports multiple engine backends, including vLLM, TensorRT-LLM, and SGLang. It disaggregates the prefill (compute-bound) and decode (memory-bound) phases across separate GPUs, enabling independent scaling and phase-specific parallelism strategies. For example, the memory-bound decode phase can leverage wide expert parallelism (EP) without constraining the compute-heavy prefill phase, improving overall resource utilization and performance.

Dynamo includes an SLA-based Planner that proactively manages GPU scaling for prefill/decode (PD) disaggregated inference. Using pre-deployment profiling, it evaluates how model parallelism and batching affect performance, recommending configurations that meet latency targets like Time to First Token (TTFT) and Inter-Token Latency (ITL) within a given GPU budget. At runtime, the Planner forecasts traffic with time-series models, dynamically adjusting PD worker counts based on predicted demand and real-time metrics.

The Dynamo LLM-aware Router manages the key-value (KV) cache across large GPU clusters by hashing requests and tracking cache locations. It calculates overlap scores between incoming requests and cached KV blocks, routing requests to GPUs that maximize cache reuse while balancing workload. This cache-aware routing reduces costly KV recomputation and avoids bottlenecks, which in turn improves performance, especially for large models with long context windows.

To reduce GPU memory overhead, the Dynamo KV Block Manager offloads infrequently accessed KV blocks to CPU RAM, SSDs, or object storage. It supports hierarchical caching and intelligent eviction policies across nodes, scaling cache storage to petabyte levels while preserving reuse efficiency.

Dynamo’s disaggregated execution model is especially effective for large, dynamic inference workloads where compute and memory demands shift across phases. The Azure Research paper "Splitwise: Efficient generative LLM inference using phase splitting" demonstrated the benefits of separating the compute-intensive prefill and memory-bound decode phases of LLM inference onto different hardware. We will explore this disaggregated model in detail in an upcoming blog post.

How Dynamo can optimize AI product recommendations in e-commerce apps​

Let’s put Dynamo’s features in context by walking through a realistic app scenario and explore how its framework addresses common inference challenges on AKS.

Imagine you operate a large e-commerce platform (or provide infrastructure for one), where customers browse thousands of products in real time. The app runs on AKS and experiences traffic surges during sales, launches, and seasonal events. The app also leverages LLMs to generate natural language outputs, such as:

• Context-aware product recommendations
• Dynamic product descriptions
• AI-generated upsells based on behavior, reviews, or search queries

This architecture powers user experiences like: “Customers who viewed this camera also looked at these accessories, chosen for outdoor use and battery compatibility.” Personalized product copies are dynamically rewritten for different segments, such as “For photographers” vs. “For frequent travelers.”

Behind the scenes, it requires a multi-stage LLM pipeline: retrieving product/user context, running prompted inference, and generating natural language outputs per session.

Common pain points and how Dynamo tackles them​

1. Heavy Prefill + Lightweight Decode = GPU Waste

   Generating personalized recommendations requires a heavy prefill stage (processing more than 8,000 tokens of context) but results in short outputs (~50 tokens). Running both on a single GPU can be inefficient.

   Dynamo Solution: The pipeline is split into two distinct stages, each deployed on separate GPUs. This allows independent configuration of GPU count and model parallelism for each phase. It also enables the use of different GPU types—for example, GPUs with high compute capability but lower memory for the prefill stage, and GPUs with both high compute and large memory capacity for the decode stage.

   In our e-commerce example, when a user lands on a product page:

   • Prefill runs uninterrupted on dedicated GPUs using model parallelism degrees optimized for accelerating math-intensive attention GEMM operation. This enables fast processing of 8,000 tokens of user context and product metadata.

   • Decode runs on a GPU pool with different counts and parallelism degrees designed and tuned to maximize memory bandwidth and capacity for generating the short product blurb.

   Result: This approach maximizes GPU utilization and reduces per-request cost.

2. Meeting SLOs and handling traffic spikes without overprovisioning

   Your SLO might define time-to-first-token < 300ms and 99th percentile latency < 500ms, but maintaining this across dynamic workloads is tough. Static GPU allocation leads to bottlenecks during traffic spikes, causing either SLO violations or wasted capacity.

   Dynamo Solution: Continuously monitors metrics and auto-scales GPU replicas or reallocates GPUs between prefill and decode stages based on real-time traffic patterns, queue depth, and latency targets.

   In our e-commerce example:

   • During Black Friday, Dynamo observes latency climbing due to a surge in prefill demand. It responds by increasing prefill GPU replicas by 50%, shifting GPUs from decode or spinning up additional ones.
   • At night, when email generation jobs dominate, Dynamo reallocates GPUs back to decode to optimize throughput.
   • When load drops, resources scale back down.

   Result: SLOs are met consistently without over or under provisioning, controlling costs while maintaining performance.

3. Recomputing shared context is wasteful

   Many requests within the same session reuse the same product or user context but unnecessarily recompute the KV cache each time, wasting valuable GPU resources that could be spent serving other user requests.

   Dynamo Solution: LLM-aware routing maintains a map of KV cache across large GPU clusters and directs requests to the GPUs that already hold the relevant KV cache, avoiding redundant computation.

   In our e-commerce example:

   • A user browses five similar items in one session.
   • Dynamo routes all requests to the same GPU that already has the user’s or product’s context cached.

   Result: Faster response times, lower latency, reduced GPU usage.

4. KV cache growth blows past GPU memory

   With many concurrent sessions and large input sequence lengths, the KV cache (product data + user history) can exceed available GPU memory. This can trigger evictions, leading to costly re-computations or inference errors.

   Dynamo Solution: The KV Block Manager (KVBM) offloads cold/unused KV cache data to CPU RAM, NVMe, or networked storage freeing valuable GPU memory for active requests.

   In our e-commerce example:

   • Without cache offloading: increasing number of concurrent sessions per GPU increases latency due to KV cache evictions and recomputations
   • With Dynamo: GPUs can support higher concurrencies while maintaining low latency

   Result: Higher concurrency at lower cost, without degrading user experience.

Enterprise-scale inference experiments: Dynamo with GB200, running on AKS​

We set out to deploy the popular open-source GPT-OSS 120B reasoning model using Dynamo on AKS on GB200 NVL72, adapting the SemiAnalysis InferenceMAX recipe for a large scale, production-grade environment.

Our approach: leverage Dynamo as the inference server and swap GB200 NVL72 nodes in place of NVIDIA HGX™ B200, scaling the deployment across multiple nodes.

Our goal was to replicate the performance results reported by SemiAnalysis, but at a larger scale within an AKS environment, proving that enterprise-scale inference with cutting-edge hardware and open-source models is not only possible, but practical.

AKS Deployment Overview​

Ready to build the same setup? Our comprehensive guide walks you through each stage of the deployment:

1. Set up your foundation: Configure GPU node pools and prepare your inference set up with the prerequisites you will need.
2. Deploy Dynamo via Helm: Get the inference server running with the right configurations for GB200 NVL72.
3. Benchmark performance with your serving engine: Test and optimize latency/throughput under production conditions.

Find the complete recipe for GPT-OSS 120B at aka.ms/dynamo-recipe-gpt-oss-120b and get hands-on with the deployment guide at aka.ms/aks-dynamo.

The results​

By following this approach, we achieved 1.2 million tokens per second, meeting our goal of replicating SemiAnalysis InferenceMAX results at enterprise scale. This demonstrates that Dynamo on AKS running on ND GB200-v6 instances can deliver the performance needed for production inference workloads.

Looking ahead​

This work reflects a deep collaboration between Azure and NVIDIA to reimagine how large-scale inference is built and operated, from the hardware up through the software stack. By combining GB200 NVL72 nodes and the open-source Dynamo project on AKS, we’ve taken a step toward making distributed inference faster, more efficient, and more responsive to real-world demands.

This post focused on the foundational serving stack. In upcoming blogs, we will build on this foundation and explore more of Dynamo's advanced features, such as Disaggregated Serving and SLA-based Planner. We'll demonstrate how these features allow for even greater efficiency, moving from a static, holistic deployment to a flexible, phase-splitted architecture. Moving forward, we also plan to extend our testing to include larger mixture-of-experts (MoE) reasoning models such as DeepSeek R1. We encourage you to try out the Dynamo recipe in this blog on AKS and share your feedback!

In this blog post, I'll show you one approach to running the AKS MCP server: deploying it inside an AKS cluster as a Streamable HTTP service. This pattern demonstrates how MCP servers can be centrally managed and made accessible to multiple clients—including AI assistants, automation tools, and even autonomous agents.

Before we get started, let's explore why you might consider this deployment pattern.

Why Deploy MCP Servers on AKS?​

Running MCP servers on AKS offers several potential advantages, depending on your requirements:

• Centralized deployment: Host multiple MCP servers in a single cluster, making them accessible to various clients without requiring local installation on each machine
• Scalability and reliability: Leverage AKS's built-in features for scaling, monitoring, logging, and high availability
• Secure authentication: Use Workload Identity for passwordless authentication with fine-grained access control
• Multi-client access: Enable different teams, applications, or autonomous agents to connect to the same MCP server instance over HTTP
• Standardized patterns: Establish a repeatable deployment pattern that can be applied to other MCP servers beyond just the AKS MCP server
• Governance and auditability: Maintain centralized control over authentication and audit trails by collecting MCP server logs in one place, providing visibility into all tool invocations across your organization

That said, this approach isn't the only way—or necessarily the best way—to run MCP servers. Local installations may be simpler for individual users, while other deployment patterns might better suit different use cases. The key is choosing the right approach for your specific requirements.

In this setup, we'll use a User-Assigned Managed Identity with Workload Identity to authenticate the MCP server to Azure, giving it necessary permissions to manage both Kubernetes and Azure resources.

Let's dive in!

Prerequisites​

Before you begin, ensure you have the following:

• Azure subscription
• Azure CLI
• kubectl
• NodeJS and npm for running MCP Inspector
• POSIX-compliant shell (e.g., bash, zsh) or use Azure Cloud Shell

With an Azure subscription and the Azure CLI installed, log into your Azure account.

az login

We'll take advantage of one preview feature of AKS so run the following command to install the Azure CLI extensions for AKS.

az extension add --name aks-preview

Register the DisableSSHPreview feature in your subscription. We don't need SSH access for this cluster so disable it at cluster creation time.

az feature register --name DisableSSHPreview --namespace Microsoft.ContainerService

Wait for the feature to be registered. You can check the status with the following command:

az feature show --name DisableSSHPreview --namespace Microsoft.ContainerService

Once the feature is registered, refresh the registration of the Microsoft.ContainerService resource provider.

az provider register --namespace Microsoft.ContainerService

With the prerequisites in place, you're ready to start setting up the AKS MCP server on AKS.

Run the following command to export a random name variable to use for resource names to avoid naming conflicts.

export RANDOM_NAME=$(petname) # If you don't have petname, replace this entire command with e.g. export RANDOM_NAME=myrandomname

Set a few environment variables to use throughout.

export LOCATION=westus3 # or any Azure region of your choice
export RESOURCE_GROUP_NAME=rg-$RANDOM_NAME
export AKS_CLUSTER_NAME=aks-$RANDOM_NAME
export MANAGED_IDENTITY_NAME=mi-$RANDOM_NAME

Provision Azure resources​

You're now ready to create the necessary Azure resources. Run the following command to create a resource group and retrieve the resource group ID. This will be used later when assigning Azure RBAC roles to the managed identity.

read -r RESOURCE_GROUP_ID <<< \
  "$(az group create \
  --name $RESOURCE_GROUP_NAME \
  --location $LOCATION \
  --query '{id:id}' -o tsv)"

For demonstration purposes, we can create a single node AKS cluster. Run the following command to create one with Workload Identity enabled and retrieve the OIDC issuer URL. This value will be used later when configuring the federated identity credential.

read -r AKS_OIDC_ISSUER_URL <<< \
  "$(az aks create \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $AKS_CLUSTER_NAME \
    --enable-workload-identity \
    --enable-oidc-issuer \
    --ssh-access disabled \
    --node-count 1 \
    --query '{oidcIssuerUrl:oidcIssuerProfile.issuerUrl}' -o tsv)"

In order to allow the MCP server to authenticate to Azure, you'll need to create a User-Assigned Managed Identity and configure it for Workload Identity.

Create a new User-Assigned Managed Identity in the same resource group as your AKS cluster and retrieve its principal ID and client ID which will be used to configure Azure role assignment and Service Account annotation.

read -r PRINCIPAL_ID CLIENT_ID <<< \
  "$(az identity create \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $MANAGED_IDENTITY_NAME \
    --query '{principalId:principalId, clientId:clientId}' -o tsv)"

Create a federated identity credential for the managed identity. This allows the Service Account in the AKS cluster to authenticate as the managed identity.

az identity federated-credential create \
  --name $MANAGED_IDENTITY_NAME \
  --identity-name $MANAGED_IDENTITY_NAME \
  --resource-group $RESOURCE_GROUP_NAME \
  --issuer $AKS_OIDC_ISSUER_URL \
  --subject system:serviceaccount:default:aks-mcp \
  --audiences api://AzureADTokenExchange

Grant the managed identity the Contributor role for the resource group so it can manage resources associated with the cluster. This includes permissions to manage the AKS cluster itself as well as any other resources in the resource group such as monitoring, networking, and storage resources.

az role assignment create \
  --assignee $PRINCIPAL_ID \
  --role "Contributor" \
  --scope $RESOURCE_GROUP_ID