For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Route AI inference requests to LLM workloads using the Kubernetes Gateway API Inference Extension.

Use agentgateway with the Kubernetes Gateway API Inference Extension and llm-d Router to route inference requests to Large Language Model (LLM) workloads in your Kubernetes environment.

The Gateway API Inference Extension defines the InferencePool API and the protocol between gateways and endpoint pickers. The llm-d Router provides a production-oriented Endpoint Picker (EPP) implementation. agentgateway routes to the InferencePool, the llm-d Router selects a model server from the pool, and agentgateway routes to that model server endpoint.

For more information, see the following resources.

About

An InferencePool groups model server pods into a routable Gateway API backend. Its endpointPickerRef identifies the llm-d Router EPP that selects a pod for each request.

Agentgateway implements the gateway side of the Inference Extension protocol. The following diagram shows the request flow.

    graph LR
    Client --> Gateway
    Gateway --> HTTPRoute
    HTTPRoute --> InferencePool
    InferencePool --> EPP["llm-d Router EPP"]
    EPP --> ModelServer["model server"]
  

The EPP returns its selected endpoint to agentgateway. Agentgateway then sends the request to that model server and returns the response to the client.

Quickstart

In this quickstart, you deploy a simulated model server, the Gateway API Inference Extension CRDs, agentgateway, and the llm-d Router in Gateway mode.

1. Deploy the Qwen3 model server simulator. The llm-d-inference-sim container mimics a vLLM model server without downloading model weights or requiring GPUs.

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-qwen3-32b
     namespace: default
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: vllm-qwen3-32b
     template:
       metadata:
         labels:
           app: vllm-qwen3-32b
           inference.networking.k8s.io/engine-type: vllm
       spec:
         containers:
           - name: vllm-sim
             image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2
             args:
               - --model
               - Qwen/Qwen3-32B
               - --port
               - "8000"
               - --max-loras
               - "2"
               - --lora-modules
               - '{"name": "food-review-1"}'
             env:
               - name: POD_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.name
               - name: NAMESPACE
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
             ports:
               - containerPort: 8000
                 name: http
                 protocol: TCP
             resources:
               requests:
                 cpu: 10m
   EOF

   Verify that the simulator deployment is available.

   kubectl wait --for=condition=available --timeout=120s \
     deployment/vllm-qwen3-32b

2. Install the Gateway API and Gateway API Inference Extension Custom Resource Definitions (CRDs).

   kubectl apply --server-side -f \
     https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.3/standard-install.yaml
   
   kubectl apply -f \
     https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.6.2/manifests.yaml

3. Install agentgateway with Inference Extension support.

   helm upgrade -i --create-namespace \
     --namespace agentgateway-system \
     --version v1.7.0 \
     agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds

   helm upgrade -i \
     --namespace agentgateway-system \
     --version v1.7.0 \
     --set inferenceExtension.enabled=true \
     agentgateway oci://cr.agentgateway.dev/charts/agentgateway

4. Create an agentgateway Gateway.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: inference-gateway
     namespace: default
   spec:
     gatewayClassName: agentgateway
     listeners:
       - name: http
         port: 80
         protocol: HTTP
   EOF

   Verify that the Gateway is programmed.

   kubectl wait --for=condition=Programmed --timeout=120s \
     gateway/inference-gateway

5. Install the llm-d Router Gateway chart. The chart creates the InferencePool, the llm-d Router EPP deployment and service, and an HTTPRoute that attaches to the Gateway.

   export ROUTER_CHART_VERSION=v0.9.0
   
   helm upgrade -i vllm-qwen3-32b \
     oci://ghcr.io/llm-d/charts/llm-d-router-gateway \
     --version $ROUTER_CHART_VERSION \
     --set router.modelServers.matchLabels.app=vllm-qwen3-32b \
     --set router.epp.resources.requests.cpu=100m \
     --set router.epp.resources.requests.memory=128Mi \
     --set router.epp.resources.limits.memory=512Mi \
     --set provider.name=none \
     --set httpRoute.create=true \
     --set httpRoute.inferenceGatewayName=inference-gateway

   The none provider value prevents the chart from creating resources for a different gateway implementation. Agentgateway still processes the chart-created HTTPRoute and InferencePool.

   Verify the pool, EPP image, and route.

   kubectl get inferencepool vllm-qwen3-32b
   kubectl get deployment vllm-qwen3-32b-epp \
     -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
   kubectl get httproute vllm-qwen3-32b

   The deployment uses the released ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0 image.

6. Send a request through agentgateway.

   kubectl port-forward service/inference-gateway 8080:80

   In a separate terminal, send the request.

   curl -i http://localhost:8080/v1/completions \
     -H 'Content-Type: application/json' \
     -d '{
       "model": "Qwen/Qwen3-32B",
       "prompt": "What is the warmest city in the USA?",
       "max_tokens": 100,
       "temperature": 0.5
     }'

   The response has an HTTP 200 OK status.

Raise the endpoint picker timeout

By default, agentgateway waits up to 10 seconds for the EPP to respond to a request. If the EPP holds a request longer, such as when EPP flow control queues the request, agentgateway returns a 500 response with the external processing failed message. The FailOpen failure mode does not apply in this case, because the EPP is reachable.

To raise the limit, apply an AgentgatewayPolicy that targets the EPP Service, which the endpointPickerRef field of the InferencePool names. The policy attaches to every Gateway that routes to the InferencePool.

1. Create the policy for the EPP Service from the quickstart.

   kubectl apply -f - <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayPolicy
   metadata:
     name: epp-timeout
     namespace: default
   spec:
     targetRefs:
       - group: ""
         kind: Service
         name: vllm-qwen3-32b-epp
     backend:
       http:
         requestTimeout: 60s
   EOF

   Setting	Description
   targetRefs	The EPP Service. The llm-d Router Gateway chart names it after the Helm release, with an -epp suffix. Create the policy in the same namespace as the InferencePool.
   backend.http.requestTimeout	The maximum time to wait for the EPP to respond. The request timeout of the HTTPRoute still limits the whole request. The llm-d Router Gateway chart sets that timeout to 300s.

2. Verify that the policy is accepted and attached to the Gateway.

   kubectl get AgentgatewayPolicy epp-timeout -n default \
     -o jsonpath='{range .status.ancestors[*]}{.ancestorRef.name}{"\n"}{range .conditions[*]}  {.type}={.status} {.message}{"\n"}{end}{end}'

   Example output:

   inference-gateway
     Accepted=True Policy accepted
     Attached=True Attached to all targets

Use AI policies with InferencePools

The quickstart routes directly from an HTTPRoute to an InferencePool. Use that pattern when you only need Gateway API Inference Extension endpoint selection.

To also use agentgateway LLM features, such as token counting, token-based rate limits, guardrails, transformations, and LLM observability, route the HTTPRoute to an AgentgatewayBackend. Then, configure a custom provider on the AgentgatewayBackend that targets the InferencePool.

    graph LR
    Client --> Gateway
    Gateway --> HTTPRoute
    HTTPRoute --> AgentgatewayBackend
    AgentgatewayBackend --> InferencePool
    InferencePool --> ModelServer["model server"]
  

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: qwen-inferencepool
  namespace: default
spec:
  ai:
    provider:
      custom:
        backendRef:
          group: inference.networking.k8s.io
          kind: InferencePool
          name: vllm-qwen3-32b
        model: Qwen/Qwen3-32B
        formats:
        - type: Completions
          path: /v1/chat/completions
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: vllm-qwen3-32b
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/chat/completions
    backendRefs:
    - group: agentgateway.dev
      kind: AgentgatewayBackend
      name: qwen-inferencepool
    timeouts:
      request: 300s

The following example applies an LLM token budget to the same HTTPRoute. Because the route points to an AgentgatewayBackend with a custom provider, agentgateway can parse the LLM response usage and enforce the token limit while still using the InferencePool for endpoint selection.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: qwen-token-budget
  namespace: default
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: vllm-qwen3-32b
  traffic:
    rateLimit:
      local:
      - tokens: 1000
        unit: Minutes

For more token rate limiting details, see Rate limiting for LLMs.

For more custom provider examples, see Custom providers.

To route requests to multiple InferencePools based on the request body model field, see Multiple inference pools.