Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,7 @@ The controller uses [Kubernetes strategic merge patch](https://kubernetes.io/doc
- **Non-zero fields** from the override are applied to the base. Existing base fields that the override does not mention are left untouched.
- **Zero-valued fields** (empty string `""`, `0`, `nil`, `false`) in the override do **not** overwrite base values. This prevents a config that does not specify a port from wiping out the well-known config's port.
- **Containers** are merged by name - the `main` container from different sources merges into a single `main` container rather than creating duplicates. Note that each pod spec (`template`, `worker`, `prefill`) has its own `main` container - these are separate and do not merge with each other. The merge only happens within the same pod spec across config layers. Other list fields like `volumes` and `env` follow standard Kubernetes strategic merge patch behavior.
- **`router.route.http.spec`**: top-level fields such as `hostnames` and `parentRefs` merge onto the well-known router-route preset without dropping preset `rules`. Gateway API `rules` lists are atomic under strategic merge, so a non-empty `rules` override **replaces** the entire preset list. To overlay per-rule defaults such as timeouts onto preset rules without replacing them, use `router.route.http.ruleDefaults`. See [Custom HTTPRoute Spec](./llmisvc-configuration.md#custom-httproute-spec).

### Example: Adding Resources Without Losing Existing Fields

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -396,23 +396,48 @@ spec:
`spec` and `refs` are mutually exclusive - use `refs` to bring your own HTTPRoute, or `spec` to have the controller create one with your custom rules.
:::

`router.route.http.spec` is merged with the well-known router-route preset using the same [strategic merge](./llmisvc-config-composition.md#strategic-merge-patch-behavior) as other `LLMInferenceService` fields:

| What you set in `spec` | Result |
|------------------------|--------|
| Top-level fields only (for example `hostnames`, `parentRefs`) | Preset rules and backendRefs are preserved; your fields are added/overlaid |
| `ruleDefaults` on `router.route.http` | Defaults such as `timeouts` and `retry` overlay onto every preset rule |
| Any non-empty `rules` list | **Replaces** the entire preset Rules list — provide a complete list |

:::warning
Supplying a partial `rules` list that includes matches, filters, or backendRefs replaces all auto-generated routes. If you only need hostnames or inherited per-rule defaults, do **not** copy a full-spec example and delete fields — use the patterns below.
:::

#### Real-world Use Cases

**1. Custom Timeouts** (for long-running LLM inference):
**1. Add hostnames** (keep auto-generated routes):

```yaml
spec:
router:
route:
http:
spec:
rules:
- timeouts:
request: "300s"
backendRequest: "300s"
hostnames:
- my-svc.example.com
```

**2. Rule defaults** (for long-running LLM inference; overlays preset rules):

```yaml
spec:
router:
route:
http:
ruleDefaults:
timeouts:
request: "300s"
backendRequest: "300s"
retry:
attempts: 3
```

**2. URL Rewrite** (multi-tenant routing):
**3. Full custom Rules** (replaces auto-generated routes — include every match/backend you need):

```yaml
spec:
Expand All @@ -431,17 +456,11 @@ spec:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /v1/completions
```

**3. Service Backend** (bypass InferencePool):

```yaml
spec:
router:
route:
http:
spec:
rules:
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: my-svc-inference-pool
port: 8000
- backendRefs:
- group: ""
kind: Service
Expand Down Expand Up @@ -672,11 +691,10 @@ spec:
gateway: {}
route:
http:
spec:
rules:
- timeouts:
request: "300s"
backendRequest: "300s"
ruleDefaults:
timeouts:
request: "300s"
backendRequest: "300s"
scheduler: {}
```

Expand Down