5 posts tagged with "KAITO"

Challenge: Grounded Does Not Mean Safe​

An enterprise RAG application retrieves an internal troubleshooting document containing an API credential, an employee email address, and a confidential project name. Retrieval improves the answer, but the model may also reproduce that information, potentially token by token through a streaming API.

Deterministic scanners can block or redact recognizable risks such as credentials, personal information, invisible Unicode, and prohibited terms. Semantic risks, including unsupported answers, toxicity, topic violations, and instruction overrides, require model-based evaluation rather than pattern matching alone.

Grounding gives the model relevant evidence for its response. Guardrails enforce what content the application is allowed to return.

KAITO RAGEngine currently provides centrally managed output guardrails for standard and streaming responses, with blocking, redaction, hot reload, metrics, and structured logs. This post configures and evaluates those controls on Azure Kubernetes Service (AKS), explains cross-chunk protection, and considers future semantic checks. RAGEngine does not currently scan user input or retrieved context or provide model-based scanners.

A chat turn is one LLM call. An agent is hundreds — a reasoning loop that plans, calls a tool, reads the result, re-plans, and only sometimes stops. The cost and latency you signed up for in a demo get multiplied by that loop, and most of those calls are easy — a tool-argument fill, a yes/no gate, a short summary — that never needed a frontier model. So the question isn't "which model is best." It's "which model should answer this specific call, and how do I govern a flood of them?" That's not one decision; it's three, on different signals.

This post wires up those three decisions on AKS as a single OpenAI-compatible endpoint: a trivial agent step lands on a cheap self-hosted model (KAITO + vLLM, placed by GPU state) while a hard step escalates to a frontier model on Azure OpenAI — every call authenticated, metered, and traced. We lean on managed Azure throughout (KAITO for serving, Azure OpenAI for the frontier path, Azure Managed Prometheus and Grafana for observability); the two layers without a drop-in managed service — the semantic router and the AI gateway — run as open source on the cluster.

Kubernetes AI Toolchain Operator (KAITO) is an operator that simplifies and automates AI/ML model inference, tuning, and RAG in a Kubernetes cluster. With the recent v0.8.0 release, KAITO has introduced intelligent autoscaling for inference workloads as an alpha feature! In this blog, we'll guide you through setting up event-driven autoscaling for vLLM inference workloads.

Overview​

In this blog, we'll guide you through setting up an OpenAI API compatible inference endpoint with llm-d and integrating with retrieval- augmented generation (RAG) on AKS. This blog will showcase its value in a key finance use case: indexing the latest SEC 10-K filings for the two S&P 500 companies and querying them. We’ll also highlight the benefits of llm-d based on its architecture and its synergy with RAG.

You may have heard of the Kubernetes AI Toolchain Operator (KAITO) announced at Ignite 2023 and KubeCon Europe this year. The open source project has gained popularity in recent months by introducing a streamlined approach to AI model deployment and flexible infrastructure provisioning on Kubernetes.

With the v0.3.0 release, KAITO has expanded the supported model library to include the Phi-3 model, but the biggest (and most exciting) addition is the ability to fine-tune open-source models. Why should you be excited about fine-tuning? Well, it’s because fine-tuning is one way of giving your foundation model additional training using a specific dataset to enhance accuracy, which ultimately improves the interaction with end-users. (Another way to increase model accuracy is Retrieval-Augmented Generation (RAG), which we touch on briefly in this section, coming soon to KAITO).