diff --git a/docs/model-serving/generative-inference/llmisvc/llmisvc-agentgateway.md b/docs/model-serving/generative-inference/llmisvc/llmisvc-agentgateway.md index d7851ffc8..0f6bd777b 100644 --- a/docs/model-serving/generative-inference/llmisvc/llmisvc-agentgateway.md +++ b/docs/model-serving/generative-inference/llmisvc/llmisvc-agentgateway.md @@ -1,40 +1,43 @@ --- -title: LLMInferenceService with AgentGateway -description: How to integrate KServe LLMInferenceService with AgentGateway for LLM-aware routing, token tracking, and GenAI telemetry +title: LLMInferenceService with agentgateway +description: How to use KServe LLMInferenceService with agentgateway as an Inference Gateway and optionally apply AI policies --- -# LLMInferenceService with AgentGateway +# LLMInferenceService with agentgateway -This guide walks through integrating a KServe LLMInferenceService with [AgentGateway](https://agentgateway.dev) to enable LLM-aware routing with token tracking, GenAI telemetry (OpenTelemetry semantic conventions), and token-based rate limiting. The key mechanism is overriding the HTTPRoute `backendRef` via `LLMInferenceServiceConfig` so that KServe's auto-generated routes point to an `AgentgatewayBackend` instead of a plain `InferencePool` or `Service`. +This guide walks through using [agentgateway](https://agentgateway.dev) as an +Inference Gateway for KServe `LLMInferenceService`. agentgateway supports the +Gateway API Inference Extension, so KServe's generated `HTTPRoute` can route +directly to the standard `InferencePool` backend. An `AgentgatewayBackend` is +only needed when you want to apply AI policies that require LLM-aware +processing, such as token-based rate limiting. It can wrap the generated +`InferencePool` so that endpoint selection remains available. -## AgentGateway Overview +## agentgateway Overview -[AgentGateway](https://github.com/agentgateway/agentgateway) is a Rust-based proxy under the [AI Agent Infrastructure Foundation (AAIF)](https://aaif.io) at the Linux Foundation. It implements the Kubernetes Gateway API but is LLM-aware: it parses OpenAI chat completion requests and responses, extracts token usage, emits OpenTelemetry GenAI semantic conventions, and enforces token-based rate limits and policies. Key custom resources: +[agentgateway](https://github.com/agentgateway/agentgateway) is a Rust-based proxy under the [AI Agent Infrastructure Foundation (AAIF)](https://aaif.io) at the Linux Foundation. It implements the Kubernetes Gateway API but is LLM-aware: it parses OpenAI chat completion requests and responses, extracts token usage, emits OpenTelemetry GenAI semantic conventions, and enforces token-based rate limits and policies. Key custom resources: -- **`AgentgatewayBackend`**: Declares a backend as an LLM provider so the gateway activates its LLM pipeline (token parsing, model tracking, GenAI telemetry). +- **`InferencePool`**: Standard Gateway API Inference Extension backend supported directly by agentgateway. +- **`AgentgatewayBackend`**: Optional backend that declares an LLM provider so the gateway can apply LLM-aware processing and AI policies. - **`AgentgatewayPolicy`**: Attaches governance policies such as token-based rate limiting. -- **`HTTPRoute`**: Standard Gateway API routing — AgentGateway supports `AgentgatewayBackend` as a `backendRef` kind. +- **`HTTPRoute`**: Standard Gateway API routing that can reference either an `InferencePool` or an `AgentgatewayBackend`. -For more information, see the [AgentGateway documentation](https://agentgateway.dev/docs). +For more information, see the [agentgateway KServe integration guide](https://agentgateway.dev/docs/kubernetes/main/integrations/kserve/), the [llm-d agentgateway guide](https://llm-d.ai/docs/infrastructure/gateway/agentgateway), and the [Gateway API Inference Extension implementation list](https://gateway-api-inference-extension.sigs.k8s.io/implementations/gateways/#gateway-implementations). -## Why This Integration Matters +## Choose a Backend Type -When KServe generates HTTPRoutes for an LLMInferenceService, the `backendRef` defaults to `InferencePool` (or `Service` for catch-all routes). AgentGateway treats these as generic HTTP traffic — it routes requests correctly but cannot activate LLM-aware features because it doesn't know the backend serves LLM traffic. +When the managed scheduler is enabled, KServe generates `HTTPRoute` resources +with an `InferencePool` backend. agentgateway supports this standard backend +without any route override. Use `AgentgatewayBackend` only when an AI policy +needs agentgateway to parse the LLM request or response. -By overriding the `backendRef` to use `AgentgatewayBackend`, the gateway recognizes the backend as an LLM provider and activates: - -| Capability | Plain Service / InferencePool | AgentgatewayBackend | -|---|---|---| -| Routing | Yes | Yes | -| Protocol detection | `http` (generic) | `llm` (LLM-aware) | -| Token parsing | No | Yes (extracts `usage.input_tokens`, `usage.output_tokens` from response body) | -| Model tracking | No | Yes (`gen_ai.request.model`, `gen_ai.response.model`) | -| GenAI semantic conventions (OTel) | No | Yes (`gen_ai.*` attributes per [OTel GenAI spec](https://opentelemetry.io/docs/specs/semconv/gen-ai/)) | -| Token-based rate limiting | No | Yes | -| Cost tracking | No | Yes | +| Backend | Use it for | Route override required | +| --- | --- | --- | +| `InferencePool` | Standard inference routing through the Gateway API Inference Extension | No | +| `AgentgatewayBackend` | AI policies such as token-based rate limiting, plus LLM-aware telemetry and model tracking | Yes | :::note -KServe supports [distributed tracing](https://github.com/kserve/kserve/pull/5481) natively via `spec.tracing`, which provides request-level spans and traces. The GenAI semantic conventions listed above are complementary — they add LLM-specific attributes (token counts, model name, operation type) at the gateway level. +KServe supports [distributed tracing](https://github.com/kserve/kserve/pull/5481) natively via `spec.tracing`, which provides request-level spans and traces. The LLM-aware telemetry available through `AgentgatewayBackend` is complementary — it adds LLM-specific attributes such as token counts, model name, and operation type at the gateway level. ::: ## Prerequisites @@ -42,12 +45,77 @@ KServe supports [distributed tracing](https://github.com/kserve/kserve/pull/5481 Before you begin, ensure you have the following components installed and configured: - A Kubernetes cluster with [KServe with Gateway API enabled](../../../admin-guide/kubernetes-deployment.md) -- [AgentGateway](https://agentgateway.dev/docs/kubernetes/latest/getting-started/) installed in your cluster -- [Gateway API CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed +- The [Gateway API CRDs](https://gateway-api.sigs.k8s.io/guides/) installed - [LLMInferenceService dependencies](./llmisvc-dependencies.md) installed - The `kubectl` command-line tool installed and configured to access your cluster - Basic understanding of [KServe concepts](../../../concepts/index.md) and [LLMInferenceService](./llmisvc-overview.md) +Configure the KServe `LLMInferenceService` controller to attach generated +routes to the shared agentgateway Gateway. During the GAIE v1 migration, KServe +also installs transitional CRDs that its controller uses for compatibility: + +```shell +export KSERVE_VERSION=v0.20.0-rc0 + +helm upgrade -i kserve-llmisvc-resources \ + oci://ghcr.io/kserve/charts/kserve-llmisvc-resources \ + --version $KSERVE_VERSION \ + --namespace kserve \ + --set kserve.controller.deploymentMode=Standard \ + --set kserve.controller.gateway.ingressGateway.enableGatewayApi=true \ + --set kserve.controller.gateway.ingressGateway.createGateway=false \ + --set kserve.controller.gateway.ingressGateway.kserveGateway=kserve/kserve-ingress-gateway \ + --set kserve.controller.gateway.ingressGateway.className=agentgateway \ + --set kserve.controller.gateway.disableIstioVirtualHost=true \ + --set kserve.controller.gateway.disableIngressCreation=false \ + --set kserve.controller.knativeAddressableResolver.enabled=false \ + --set kserve.controller.gateway.localGateway.gateway="" \ + --set kserve.controller.gateway.localGateway.gatewayService="" +``` + +Wait for the updated controller, then apply the final GAIE v1.5.0 CRD bundle. +Applying the bundle after the KServe chart updates the stable API definitions +and retains KServe's transitional CRDs: + +```shell +kubectl rollout status deployment/llmisvc-controller-manager \ + --namespace kserve \ + --timeout=240s + +kubectl apply --server-side -f \ + https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml +``` + +Install or upgrade agentgateway after the GAIE CRDs so that its controller +discovers `InferencePool`, then install the matching KServe runtime +configuration: + +```shell +export AGENTGATEWAY_VERSION=v1.4.1 + +helm upgrade -i agentgateway-crds \ + oci://cr.agentgateway.dev/charts/agentgateway-crds \ + --version $AGENTGATEWAY_VERSION \ + --namespace agentgateway-system \ + --create-namespace + +helm upgrade -i agentgateway \ + oci://cr.agentgateway.dev/charts/agentgateway \ + --version $AGENTGATEWAY_VERSION \ + --namespace agentgateway-system \ + --set inferenceExtension.enabled=true + +helm upgrade -i kserve-runtime-configs \ + oci://ghcr.io/kserve/charts/kserve-runtime-configs \ + --version $KSERVE_VERSION \ + --namespace kserve \ + --set kserve.llmisvcConfigs.enabled=true +``` + +KServe creates the `InferencePool` and deploys the llm-d Router endpoint picker +from its runtime configuration. Do not install the llm-d Router Helm chart +separately for this workflow. + ## Deploy LLMInferenceService ### Create Namespace @@ -58,14 +126,15 @@ kubectl create namespace kserve-test ### Create Gateway -Create an AgentGateway Gateway resource: +Create a shared agentgateway Gateway resource in the `kserve` namespace. +Routes from model namespaces can attach to this Gateway: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: - name: agentgateway - namespace: kserve-test + name: kserve-ingress-gateway + namespace: kserve spec: gatewayClassName: agentgateway listeners: @@ -74,7 +143,10 @@ spec: port: 80 allowedRoutes: namespaces: - from: Same + from: All + infrastructure: + labels: + serving.kserve.io/gateway: kserve-ingress-gateway ``` ### Deploy Your Model @@ -82,7 +154,7 @@ spec: Deploy an LLMInferenceService. This example uses a small model for demonstration; replace with your model of choice: ```yaml -apiVersion: serving.kserve.io/v1alpha1 +apiVersion: serving.kserve.io/v1alpha2 kind: LLMInferenceService metadata: name: my-model @@ -94,6 +166,7 @@ spec: replicas: 1 router: route: {} + scheduler: {} template: containers: - name: main @@ -111,11 +184,35 @@ kubectl wait --for=condition=Ready llminferenceservice/my-model \ -n kserve-test --timeout=300s ``` -## Configure LLM-Aware Routing +## Use the Standard InferencePool Backend + +The managed scheduler and default +[LLMInferenceServiceConfig route template](https://github.com/kserve/kserve/blob/master/config/llmisvcconfig/config-llm-router-route.yaml) +generate an `InferencePool` and an `HTTPRoute` that references it. agentgateway +supports this backend directly, so no `AgentgatewayBackend` or route override +is required for standard inference routing. + +Verify the generated backend reference: + +```shell +kubectl get httproute my-model-kserve-route \ + -n kserve-test \ + -o jsonpath='{.spec.rules[?(@.name=="v1-chat-completions-path")].backendRefs[0]}' +``` + +If you do not need AI policies, continue to [Configure the gateway URL](#configure-gateway-url). + +## Optional: Configure an AgentgatewayBackend for AI Policies + +To use AI policies that require LLM-aware request or response processing, +create an `AgentgatewayBackend` and override the generated route to reference +it. ### Step 1: Create AgentgatewayBackend -Create an `AgentgatewayBackend` that declares the KServe predictor Service as an LLM provider. This tells AgentGateway to activate its LLM pipeline for traffic to this backend: +Create an `AgentgatewayBackend` that wraps the `InferencePool` generated by +KServe. This tells agentgateway to activate its LLM pipeline while retaining +the pool's endpoint selection: ```yaml apiVersion: agentgateway.dev/v1alpha1 @@ -126,17 +223,23 @@ metadata: spec: ai: provider: - openai: + custom: + backendRef: + group: inference.networking.k8s.io + kind: InferencePool + name: my-model-inference-pool model: Qwen/Qwen2.5-0.5B-Instruct - host: my-model-kserve-workload-svc.kserve-test.svc.cluster.local - port: 8000 + formats: + - type: Completions + path: /v1/chat/completions ``` :::tip -The `host` should point to the KServe workload Service. The naming convention is `{llminferenceservice-name}-kserve-workload-svc`. Verify with: +The generated pool name is `{llminferenceservice-name}-inference-pool`. +Verify it with: ```shell -kubectl get svc -n kserve-test | grep workload +kubectl get inferencepool -n kserve-test ``` ::: @@ -153,7 +256,7 @@ import TabItem from '@theme/TabItem'; Override the route configuration on an individual `LLMInferenceService` using `spec.router.route.http`: ```yaml -apiVersion: serving.kserve.io/v1alpha1 +apiVersion: serving.kserve.io/v1alpha2 kind: LLMInferenceService metadata: name: my-model @@ -164,14 +267,15 @@ spec: name: Qwen/Qwen2.5-0.5B-Instruct replicas: 1 router: + scheduler: {} route: http: spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway - name: agentgateway - namespace: kserve-test + name: kserve-ingress-gateway + namespace: kserve rules: - backendRefs: - group: agentgateway.dev @@ -211,7 +315,7 @@ spec: Create an `LLMInferenceServiceConfig` that overrides the route template. This can be referenced by multiple `LLMInferenceService` resources via `baseRefs`: ```yaml -apiVersion: serving.kserve.io/v1alpha1 +apiVersion: serving.kserve.io/v1alpha2 kind: LLMInferenceServiceConfig metadata: name: agentgateway-route-config @@ -224,8 +328,8 @@ spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway - name: agentgateway - namespace: kserve-test + name: kserve-ingress-gateway + namespace: kserve rules: - backendRefs: - group: agentgateway.dev @@ -254,7 +358,7 @@ spec: Then reference it in your LLMInferenceService: ```yaml -apiVersion: serving.kserve.io/v1alpha1 +apiVersion: serving.kserve.io/v1alpha2 kind: LLMInferenceService metadata: name: my-model @@ -266,6 +370,8 @@ spec: replicas: 1 baseRefs: - name: agentgateway-route-config + router: + scheduler: {} template: containers: - name: main @@ -279,7 +385,7 @@ spec: -### Step 3: Attach Token-Based Rate Limiting (Optional) +### Step 3: Attach Token-Based Rate Limiting Apply an `AgentgatewayPolicy` to enforce token-based rate limits on the route: @@ -293,7 +399,7 @@ spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute - name: my-model-llminferenceservice-route + name: my-model-kserve-route traffic: rateLimit: local: @@ -301,12 +407,13 @@ spec: unit: Hours ``` -## Configure $GATEWAY_URL +## Configure $GATEWAY_URL {#configure-gateway-url} Check if your Gateway has an external IP address assigned: ```shell -kubectl get svc -n kserve-test -l gateway.networking.k8s.io/gateway-name=agentgateway +kubectl get svc -n kserve \ + -l gateway.networking.k8s.io/gateway-name=kserve-ingress-gateway ``` @@ -314,7 +421,7 @@ kubectl get svc -n kserve-test -l gateway.networking.k8s.io/gateway-name=agentga If the EXTERNAL-IP shows an actual IP address (not <pending>): ```shell - export GATEWAY_URL="http://$(kubectl get gateway -n kserve-test agentgateway \ + export GATEWAY_URL="http://$(kubectl get gateway -n kserve kserve-ingress-gateway \ -o jsonpath='{.status.addresses[0].value}')" ``` @@ -323,17 +430,36 @@ kubectl get svc -n kserve-test -l gateway.networking.k8s.io/gateway-name=agentga ```shell export GATEWAY_URL="http://localhost:8080" - kubectl port-forward -n kserve-test svc/agentgateway 8080:80 + kubectl port-forward -n kserve svc/kserve-ingress-gateway 8080:80 ``` ## Testing the Integration +Set the request path for the backend you selected: + + + + +```shell +export GATEWAY_PATH="/kserve-test/my-model/v1/chat/completions" +``` + + + + +```shell +export GATEWAY_PATH="/v1/chat/completions" +``` + + + + Send a test request: ```shell -curl -s "$GATEWAY_URL/v1/chat/completions" \ +curl -s "$GATEWAY_URL$GATEWAY_PATH" \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-0.5B-Instruct", @@ -341,18 +467,19 @@ curl -s "$GATEWAY_URL/v1/chat/completions" \ }' | jq . ``` -### Verify LLM-Aware Processing +### Verify AI Policy Processing -Check the AgentGateway logs to confirm the LLM pipeline is active: +If you configured an `AgentgatewayBackend`, check the agentgateway logs to +confirm the LLM pipeline is active: ```shell -kubectl logs -n kserve-test deploy/agentgateway --tail=10 +kubectl logs -n kserve deploy/kserve-ingress-gateway --tail=10 ``` With `AgentgatewayBackend`, you should see GenAI fields in the log: ```text -route=my-model-llminferenceservice-route +route=kserve-test/my-model-kserve-route http.status=200 protocol=llm gen_ai.operation.name=chat @@ -362,28 +489,26 @@ gen_ai.usage.input_tokens=12 gen_ai.usage.output_tokens=15 ``` -Without `AgentgatewayBackend` (plain Service or InferencePool backendRef), the same request would only show: - -```text -protocol=http -http.status=200 -``` - ## How It Works -The default [LLMInferenceServiceConfig route template](https://github.com/kserve/kserve/blob/master/config/llmisvcconfig/config-llm-router-route.yaml) generates HTTPRoutes with `backendRef` pointing to `InferencePool` (kind: `InferencePool`, group: `inference.networking.x-k8s.io`). - -By overriding `spec.router.route.http` — either directly on the `LLMInferenceService` or via a reusable `LLMInferenceServiceConfig` — you change the `backendRef` to `AgentgatewayBackend` (kind: `AgentgatewayBackend`, group: `agentgateway.dev`). This leverages Gateway API's support for [arbitrary backendRef kinds (GEP-1742)](https://gateway-api.sigs.k8s.io/geps/gep-1742/). +For standard inference routing, KServe generates an `HTTPRoute` whose +`backendRef` points to `InferencePool` (kind: `InferencePool`, group: +`inference.networking.k8s.io`). agentgateway implements the Gateway API +Inference Extension and routes this traffic through the pool's endpoint picker. -AgentGateway recognizes `AgentgatewayBackend` references and activates its LLM pipeline: parsing OpenAI request/response payloads, extracting token usage, emitting GenAI telemetry, and enforcing token-based policies. - -:::note -This approach is generic. Any gateway that supports custom `backendRef` kinds via Gateway API can use the same `LLMInferenceServiceConfig` override mechanism. Replace `AgentgatewayBackend` with your gateway's backend CRD. -::: +For AI policies, overriding `spec.router.route.http` changes the `backendRef` +to `AgentgatewayBackend` (kind: `AgentgatewayBackend`, group: +`agentgateway.dev`). The `AgentgatewayBackend` wraps the generated +`InferencePool`, so agentgateway can parse OpenAI request and response payloads, +extract token usage, emit GenAI telemetry, and enforce token-based policies +without bypassing the endpoint picker. ## Next Steps -- Explore the [AgentGateway documentation](https://agentgateway.dev/docs) for advanced features like content filtering and cost tracking. +- Compare the other [inference gateway integrations](./llmisvc-inference-gateways.md). +- Follow the [llm-d agentgateway guide](https://llm-d.ai/docs/infrastructure/gateway/agentgateway) for the llm-d Router and gateway infrastructure. +- Explore the [llm-d well-lit paths](https://llm-d.ai/docs/well-lit-paths) for production deployment patterns. +- Explore the [agentgateway documentation](https://agentgateway.dev/docs) for advanced features like content filtering and cost tracking. - Learn more about [LLMInferenceServiceConfig composition](./llmisvc-config-composition.md) for managing configurations across multiple services. - See the [LLMInferenceService Configuration Guide](./llmisvc-configuration.md) for the full `spec.router.route.http` reference. - Follow the discussion in [kserve/kserve#5729](https://github.com/kserve/kserve/issues/5729) and [agentgateway/agentgateway#2323](https://github.com/agentgateway/agentgateway/issues/2323) for ongoing integration improvements. diff --git a/docs/model-serving/generative-inference/llmisvc/llmisvc-config-composition.md b/docs/model-serving/generative-inference/llmisvc/llmisvc-config-composition.md index 62f058240..297dc549a 100644 --- a/docs/model-serving/generative-inference/llmisvc/llmisvc-config-composition.md +++ b/docs/model-serving/generative-inference/llmisvc/llmisvc-config-composition.md @@ -100,7 +100,7 @@ The sources that get merged (well-known configs are auto-injected, baseRef is re
kserve-config-llm-scheduler
-{"router:\n  scheduler:\n    pool:\n      spec:\n        selector: [...]\n        targetPort: 8000\n    template:\n      containers:\n        - name: epp\n          image: llm-d-inference-scheduler\n          ports: [9002]\n        - name: tokenizer\n          image: llm-d-uds-tokenizer\n          ports: [8082]"}
+{"router:\n  scheduler:\n    pool:\n      spec:\n        selector: [...]\n        targetPorts:\n          - number: 8000\n    template:\n      containers:\n        - name: epp\n          image: llm-d-router-endpoint-picker\n          ports: [9002]\n        - name: tokenizer\n          image: llm-d-uds-tokenizer\n          ports: [8082]"}
@@ -199,10 +199,10 @@ The sources that get merged (well-known configs are auto-injected, baseRef is re {" template:\n"} {" containers:\n"} {" - name: epp\n"} -{" image: llm-d-inference-scheduler\n"} +{" image: llm-d-router-endpoint-picker\n"} {" ports: [9002]\n"} {" - name: tokenizer\n"} -{" image: llm-d-inference-scheduler\n"} +{" image: llm-d-router-endpoint-picker\n"} {" ports: [8082]\n"} {" route:\n"} {" http:\n"} diff --git a/docs/model-serving/generative-inference/llmisvc/llmisvc-dependencies.md b/docs/model-serving/generative-inference/llmisvc/llmisvc-dependencies.md index 938953439..b3cc4904e 100644 --- a/docs/model-serving/generative-inference/llmisvc/llmisvc-dependencies.md +++ b/docs/model-serving/generative-inference/llmisvc/llmisvc-dependencies.md @@ -50,7 +50,6 @@ Extends Gateway API with **LLM-specific scheduling and load balancing** capabili ### Why Required - **Intelligent Routing**: Routes requests to optimal pods based on KV cache, load, and prefill-decode separation - **InferencePool**: Represents a pool of inference pods with custom scheduling logic -- **InferenceModel**: Defines model metadata and criticality for scheduling decisions ### How LLMInferenceService Uses It @@ -58,7 +57,14 @@ Extends Gateway API with **LLM-specific scheduling and load balancing** capabili LLMInferenceService creates GIE resources when scheduler is enabled. -**Key Point**: InferencePool has `extensionRef` pointing to the EPP (Endpoint Picker) service, which runs the scheduling logic. +**Key Point**: `InferencePool` has an `endpointPickerRef` that points to the +llm-d Router service, which runs the scheduling logic. + +For current production guidance, see the +[llm-d well-lit paths](https://llm-d.ai/docs/well-lit-paths), the +[llm-d Router gateway guides](https://llm-d.ai/docs/infrastructure/gateway), +and the +[Gateway API CRD installation guide](https://llm-d.ai/docs/infrastructure/gateway/install-crds). --- @@ -145,7 +151,7 @@ Gateway API is a **specification**, not an implementation. The actual traffic ro 1. **Gateway Provider Initialization**: When a Gateway Provider (e.g., Envoy Gateway, Istio) starts, it scans for available CRDs to determine which extensions to support. -2. **GIE Support**: If GIE CRDs are installed **after** the Gateway Provider, the provider won't know about `InferencePool` and `InferenceModel` resources. +2. **GIE Support**: If GIE CRDs are installed **after** the Gateway Provider, the provider might not detect `InferencePool` resources. 3. **Operator Restart Required**: Installing GIE CRDs later requires restarting the Gateway Provider operator to detect the new CRDs. @@ -153,17 +159,17 @@ Gateway API is a **specification**, not an implementation. The actual traffic ro ```bash # Step 1: Install cert-manager (required by LWS) -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.0/cert-manager.yaml +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml # Step 2: Install Gateway API CRDs -kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml +kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml # Step 3: Install GIE CRDs (BEFORE Gateway Provider!) -kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/install.yaml +kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml # Step 4: Install Gateway Provider (Envoy Gateway example) helm install eg oci://docker.io/envoyproxy/gateway-helm \ - --version v1.2.4 \ + --version v1.8.3 \ -n envoy-gateway-system --create-namespace # Step 5: Install LWS Operator (if using multi-node) @@ -306,4 +312,3 @@ curl http://localhost:8080/v1/completions ... | **Prefill-Decode** | ✅ | ✅ (required) | ❌ (single-node) or ✅ (multi-node) | ✅ | | **DP+EP** | ✅ | ✅ (scheduler) | ✅ | ✅ | | **No Scheduler** | ✅ | ❌ | ❌ | ✅ | - diff --git a/docs/model-serving/generative-inference/llmisvc/llmisvc-envoy-ai-gateway.md b/docs/model-serving/generative-inference/llmisvc/llmisvc-inference-gateway-extension.md similarity index 87% rename from docs/model-serving/generative-inference/llmisvc/llmisvc-envoy-ai-gateway.md rename to docs/model-serving/generative-inference/llmisvc/llmisvc-inference-gateway-extension.md index ef122405b..459edf17f 100644 --- a/docs/model-serving/generative-inference/llmisvc/llmisvc-envoy-ai-gateway.md +++ b/docs/model-serving/generative-inference/llmisvc/llmisvc-inference-gateway-extension.md @@ -1,11 +1,22 @@ --- -title: LLMInferenceService with Inference Gateway Extension (IGW) +title: LLMInferenceService with Envoy AI Gateway description: How to integrate KServe LLMInferenceService with Envoy AI Gateway to manage LLM traffic and usage-based rate limits --- -# LLMInferenceService with Inference Gateway Extension (IGW) - -This tutorial walks through deploying a KServe LLMInferenceService that wraps [llm-d](https://llm-d.ai/) — which implements the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) (the llm-d router and inference pool) — and fronts it with Envoy AI Gateway to provide OpenAI-compatible routing, token usage accounting, and usage-based rate limiting. KServe integrates with llm-d via a Kubernetes-native custom resource, LLMInferenceService, which provisions the router and inference pool. You will create a Gateway and an AIGatewayRoute that forward requests to the KServe InferencePool, enable automatic token metering (input, output, and total) via llmRequestCosts, and enforce per-user, per-model quotas using a BackendTrafficPolicy. KServe can run behind the AI Gateway in the same cluster or a different one; for clarity, this guide uses a single-cluster setup. +# LLMInferenceService with Envoy AI Gateway + +This tutorial walks through deploying a KServe LLMInferenceService that uses +the [llm-d Router](https://llm-d.ai/docs/infrastructure/gateway) with +[Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) +resources, and fronts it with Envoy AI Gateway to provide OpenAI-compatible +routing, token usage accounting, and usage-based rate limiting. KServe +integrates with llm-d via a Kubernetes-native custom resource, +LLMInferenceService, which provisions the router and inference pool. You will +create a Gateway and an AIGatewayRoute that forward requests to the KServe +InferencePool, enable automatic token metering (input, output, and total) via +llmRequestCosts, and enforce per-user, per-model quotas using a +BackendTrafficPolicy. KServe can run behind the AI Gateway in the same cluster +or a different one; for clarity, this guide uses a single-cluster setup. ## AI Gateway Overview @@ -21,7 +32,7 @@ For more information, see the [Envoy AI Gateway documentation](https://aigateway ## llm-d Overview -[llm-d](https://llm-d.ai/) is a Kubernetes-native distributed inference serving stack, providing well-lit paths for anyone to serve large generative AI models at scale, with the fastest time-to-value and competitive performance per dollar for most models across most hardware accelerators. +[llm-d](https://llm-d.ai/) is a Kubernetes-native distributed inference serving stack, providing [well-lit paths](https://llm-d.ai/docs/well-lit-paths) for anyone to serve large generative AI models at scale, with the fastest time-to-value and competitive performance per dollar for most models across most hardware accelerators. KServe's generative inference leverages llm-d components to scale and schedule traffic efficiently: @@ -39,7 +50,7 @@ In this tutorial you'll deploy an `LLMInferenceService` that creates a router an Before you begin, ensure you have the following components installed and configured: - A Kubernetes cluster with [KServe with Gateway API Enabled](../../../admin-guide/kubernetes-deployment.md) -- [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/guides/) installed in your cluster +- [Gateway API and Gateway API Inference Extension CRDs](https://llm-d.ai/docs/infrastructure/gateway/install-crds) installed in your cluster - [Envoy Gateway with Inference Pool support enabled prerequisites](https://aigateway.envoyproxy.io/docs/getting-started/prerequisites) installed in your cluster - [Envoy AI Gateway](https://aigateway.envoyproxy.io/docs/getting-started/installation) installed in your cluster - [LeaderWorkerSet (LWS)](https://lws.sigs.k8s.io/docs/installation/) installed in your cluster @@ -98,7 +109,10 @@ spec: ### Create EndpointPickerConfig -The Endpoint Picker (EPP) or scheduler is a core component of the Gateway API Inference Extension. It is responsible for selecting the best backend endpoint (pod) from the InferencePool for each request. You can customize the scheduling behavior by defining various plugins for scoring, filtering, and picking endpoints based on your requirements. +The llm-d Router provides the Endpoint Picker (EPP) used with the Gateway API +Inference Extension. The EPP selects the best backend endpoint from the +`InferencePool` for each request. You can customize its behavior with plugins +for scoring, filtering, and selecting endpoints. About the configuration: @@ -109,13 +123,11 @@ About the configuration: Common plugins used in this guide: -- single-profile-handler (Profile Handler): Always selects a single, primary profile. Parameters: none. -- prefix-cache-scorer (Scorer): Increases score for pods likely to contain more of the prompt in their KV cache, improving latency and throughput. Parameters: - - hashBlockSize: Block size for prompt hashing (default: 64). - - maxPrefixBlocksToMatch: Maximum number of prefix blocks to match (default: 256). - - lruCapacityPerServer: LRU index capacity per server/pod (default: 31250). -- load-aware-scorer (Scorer): Scores candidates based on current load; lower load yields a higher score. Parameters may include sensitivity controls such as threshold (example below uses threshold: 100). -- max-score-picker (Picker): Chooses the candidate with the highest aggregate score. +- queue-scorer: Prefers endpoints with shorter request queues. +- kv-cache-utilization-scorer: Accounts for each endpoint's KV-cache usage. +- prefix-cache-scorer: Prefers endpoints that already cache more of the request prefix. +- metrics-data-source: Reads model server metrics used by scoring plugins. +- core-metrics-extractor: Extracts the core vLLM metrics used by the router. ```yaml apiVersion: v1 @@ -125,23 +137,27 @@ metadata: namespace: kserve-test data: endpoint-picker-config.yaml: | - apiVersion: inference.networking.x-k8s.io/v1alpha1 + apiVersion: llm-d.ai/v1alpha1 kind: EndpointPickerConfig plugins: - - type: single-profile-handler + - type: queue-scorer + - type: kv-cache-utilization-scorer - type: prefix-cache-scorer - - type: load-aware-scorer + - type: metrics-data-source parameters: - threshold: 100 - - type: max-score-picker + scheme: http + path: /metrics + insecureSkipVerify: true + - type: core-metrics-extractor schedulingProfiles: - name: default plugins: + - pluginRef: queue-scorer + weight: 2 + - pluginRef: kv-cache-utilization-scorer + weight: 2 - pluginRef: prefix-cache-scorer - weight: 2.0 - - pluginRef: load-aware-scorer - weight: 1.0 - - pluginRef: max-score-picker + weight: 3 ``` ### Create LLMInferenceServiceConfig @@ -150,7 +166,7 @@ In this step, you’ll define an LLMInferenceServiceConfig — a reusable templa In this example, we will configure: - vLLM worker defaults: image, command/args that pass the served model name, port 8000, logging level, HF cache path, liveness/readiness probes, secure pod settings, and volumes for /home, /dev/shm, model cache, and TLS certs. -- Router and scheduler defaults: an inference scheduler (gRPC + metrics ports) configured for secure serving and wired to the EndpointPickerConfig (from the ConfigMap above) to score/pick endpoints; the pool targets port 8000 and references an internal EPP service. +- Router and scheduler defaults: an inference scheduler (gRPC + metrics ports) configured for secure serving and wired to the EndpointPickerConfig (from the ConfigMap above) to score/pick endpoints; the pool targets port 8000 and references an internal llm-d Router service. - Operational safeguards: conservative timeouts and termination grace, plus readiness/liveness for safe rollouts. @@ -242,13 +258,16 @@ spec: scheduler: pool: spec: - extensionRef: + endpointPickerRef: failureMode: FailOpen kind: Service name: |- {{ ChildName .ObjectMeta.Name `-epp-service` }} + port: + number: 9002 selector: { } - targetPortNumber: 8000 + targetPorts: + - number: 8000 template: containers: - name: main @@ -262,13 +281,13 @@ spec: - containerPort: 9090 name: metrics protocol: TCP - image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.2.0 + image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 grpc: port: 9003 - service: envoy.service.ext_proc.v3.ExternalProcessor + service: liveness initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 @@ -277,26 +296,29 @@ spec: failureThreshold: 3 grpc: port: 9003 - service: envoy.service.ext_proc.v3.ExternalProcessor + service: readiness initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 - args: - - --poolName + command: + - /app/epp + - --pool-name - "{{ ChildName .ObjectMeta.Name `-inference-pool` }}" - - --poolNamespace + - --pool-namespace - "{{ .ObjectMeta.Namespace }}" - --zap-encoder - json - - --grpcPort + - --grpc-port - "9002" - - --grpcHealthPort + - --grpc-health-port - "9003" - - --secureServing - - --certPath + - --secure-serving=true + - --enable-cert-reload=true + - --model-server-metrics-scheme=https + - --cert-path - "/etc/ssl/certs" - - --configFile + - --config-file - "/etc/config/endpoint-picker-config.yaml" resources: requests: @@ -399,7 +421,7 @@ spec: name: x-ai-eg-model value: Qwen/Qwen2.5-0.5B-Instruct backendRefs: - - group: inference.networking.x-k8s.io + - group: inference.networking.k8s.io kind: InferencePool name: qwen-instruct-inference-pool # Route to the InferencePool created by the LLMInferenceService timeouts: @@ -636,6 +658,11 @@ done Now that you've tested the basic setup, you can: +- Compare the other [inference gateway integrations](./llmisvc-inference-gateways.md). +- Review the [llm-d gateway guides](https://llm-d.ai/docs/infrastructure/gateway). +- Follow the [llm-d Envoy AI Gateway guide](https://llm-d.ai/docs/infrastructure/gateway/envoy-ai-gateway). +- Explore the [llm-d well-lit paths](https://llm-d.ai/docs/well-lit-paths) for production deployment patterns. + - Explore more rate limiter-related configuration at the [Envoy AI Gateway documentation](https://aigateway.envoyproxy.io/docs/capabilities/usage-based-ratelimiting). - Explore the [Envoy AI Gateway documentation](https://aigateway.envoyproxy.io/docs/) to learn more about the features and capabilities. diff --git a/docs/model-serving/generative-inference/llmisvc/llmisvc-inference-gateways.md b/docs/model-serving/generative-inference/llmisvc/llmisvc-inference-gateways.md new file mode 100644 index 000000000..afd613c75 --- /dev/null +++ b/docs/model-serving/generative-inference/llmisvc/llmisvc-inference-gateways.md @@ -0,0 +1,29 @@ +--- +title: Inference Gateways +description: Reference inference gateway integrations for KServe LLMInferenceService +--- + +# Inference Gateways + +KServe `LLMInferenceService` integrates with Kubernetes Gateway API +implementations to expose models, route inference requests, and apply +gateway-specific traffic policies. Choose a reference integration based on the +gateway and LLM traffic-management features you want to use. + +## Reference integrations + +| Gateway | Integration | Use this guide for | +| --- | --- | --- | +| [agentgateway](https://agentgateway.dev/) | [LLMInferenceService with agentgateway](./llmisvc-agentgateway.md) | Standard `InferencePool` routing with optional `AgentgatewayBackend` configuration for AI policies | +| [Envoy AI Gateway](https://aigateway.envoyproxy.io/) | [LLMInferenceService with Envoy AI Gateway](./llmisvc-inference-gateway-extension.md) | Routing to an `InferencePool` with an endpoint picker, OpenAI-compatible routing, token accounting, and usage-based rate limiting | + +Both guides use Gateway API resources and KServe's configurable router +templates. Their gateway-specific custom resources and supported policies are +different, so follow the guide for the gateway installed in your cluster. + +For production llm-d Router deployment patterns, see the +[llm-d well-lit paths](https://llm-d.ai/docs/well-lit-paths) and +[gateway guides](https://llm-d.ai/docs/infrastructure/gateway). The llm-d +documentation also provides provider-specific infrastructure guides for +[agentgateway](https://llm-d.ai/docs/infrastructure/gateway/agentgateway) and +[Envoy AI Gateway](https://llm-d.ai/docs/infrastructure/gateway/envoy-ai-gateway). diff --git a/docusaurus.config.ts b/docusaurus.config.ts index 2e60799fc..a855aad51 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -538,6 +538,10 @@ const config: Config = { from: '/latest/admin/ai-gateway_integration/', to: '/docs/model-serving/generative-inference/ai-gateway/envoy-ai-gateway', }, + { + from: '/docs/next/model-serving/generative-inference/llmisvc/llmisvc-envoy-ai-gateway', + to: '/docs/next/model-serving/generative-inference/llmisvc/llmisvc-inference-gateway-extension', + }, // Predictive Inference Section { diff --git a/sidebars.ts b/sidebars.ts index 6ba20d6e1..581a5a195 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -141,9 +141,13 @@ const sidebars: SidebarsConfig = { "model-serving/generative-inference/llmisvc/llmisvc-dependencies", { type: 'category', - label: 'Integrations', + label: 'Inference Gateways', + link: { + type: 'doc', + id: 'model-serving/generative-inference/llmisvc/llmisvc-inference-gateways', + }, items: [ - "model-serving/generative-inference/llmisvc/llmisvc-envoy-ai-gateway", + "model-serving/generative-inference/llmisvc/llmisvc-inference-gateway-extension", "model-serving/generative-inference/llmisvc/llmisvc-agentgateway", ], },