feat(chart): add opt-in enableInTreeAutoscaling support

README.md

@@ -214,11 +214,15 @@ Key values in `chart/values.yaml`:
 | `head.resources.requests.memory` | `2Gi` | Head node memory request |
 | `head.runtimeClassName` | - | Runtime class for head pod (e.g., `nvidia` for GPU) |
 | `worker.replicas` | `1` | Number of worker nodes |
-| `worker.minReplicas` | `1` | Min workers (for autoscaling) |
-| `worker.maxReplicas` | `1` | Max workers (for autoscaling) |
+| `worker.minReplicas` | `1` | Lower bound on workers; the autoscaler's floor when `autoscaling.enabled` |
+| `worker.maxReplicas` | `1` | Upper bound on workers; the autoscaler's ceiling when `autoscaling.enabled`. Raise it or autoscaling does nothing |
 | `worker.resources.requests.cpu` | `1` | Worker CPU request |
 | `worker.resources.requests.memory` | `2Gi` | Worker memory request |
 | `worker.runtimeClassName` | - | Runtime class for worker pods (e.g., `nvidia` for GPU) |
+| `autoscaling.enabled` | `false` | Enable Ray in-tree autoscaling of worker pods |
+| `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down |
+| `autoscaling.upscalingMode` | `Default` | `Default` or `Conservative` (rate-limited); `Aggressive` is an alias for `Default` |
+| `autoscaling.resources` | `{}` | Autoscaler sidecar resources; KubeRay's default is `500m` / `512Mi` for both requests and limits |
 
 ### Serve Applications
 

chart/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: nebari-rayserve-pack
 description: A Nebari Software Pack for Ray Serve
 type: application
-version: 0.4.1
+version: 0.5.0
 appVersion: "2.43.0"
 dependencies:
   - name: kuberay-operator

chart/templates/NOTES.txt

@@ -3,6 +3,15 @@
 The RayService controller manages the Ray cluster and Serve proxy.
 Serve is pre-initialized with host 0.0.0.0 on port 8000.
 
+{{- if .Values.autoscaling.enabled }}
+{{- if not (gt (int .Values.worker.maxReplicas) (int .Values.worker.minReplicas)) }}
+
+WARNING: autoscaling.enabled is true but the worker range is
+  [{{ .Values.worker.minReplicas }}, {{ .Values.worker.maxReplicas }}], so nothing can scale. Raise worker.maxReplicas.
+
+{{- end }}
+{{- end }}
+
 {{- if .Values.nebariapp.enabled }}
 
 NebariApp is ENABLED. The nebari-operator will configure:

chart/templates/rayservice.yaml

@@ -19,6 +19,16 @@ spec:
 
   rayClusterConfig:
     rayVersion: {{ .Values.image.tag | quote }}
+    {{- if .Values.autoscaling.enabled }}
+    enableInTreeAutoscaling: true
+    autoscalerOptions:
+      idleTimeoutSeconds: {{ .Values.autoscaling.idleTimeoutSeconds }}
+      upscalingMode: {{ .Values.autoscaling.upscalingMode | quote }}
+      {{- with .Values.autoscaling.resources }}
+      resources:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+    {{- end }}
     headGroupSpec:
       rayStartParams:
         dashboard-host: "0.0.0.0"

chart/values.yaml

@@ -162,6 +162,9 @@ head:
 # Ray Workers
 # =============================================================================
 worker:
+  # A values-only change to these does not reach a running RayCluster: KubeRay
+  # leaves them to the autoscaler. Patch the RayCluster in place instead (see
+  # docs, "Adding workers"). Chart-side fix tracked in rayserve-pack#41.
   replicas: 1
   minReplicas: 1
   maxReplicas: 1
@@ -222,6 +225,55 @@ worker:
     timeoutSeconds: 2
     failureThreshold: 120
 
+# =============================================================================
+# Ray In-Tree Autoscaling
+# =============================================================================
+# Off by default. When enabled, the chart sets `enableInTreeAutoscaling: true` and
+# KubeRay attaches the Ray autoscaler as a sidecar on the head pod. It adds
+# worker pods within [worker.minReplicas, worker.maxReplicas] when Ray tasks,
+# actors, or Serve replicas have nowhere to run, and removes pods idle for
+# idleTimeoutSeconds; worker.replicas is only the starting size.
+#
+# Treat the range as a capacity budget set once, not a tuning knob:
+#   - Both bounds default to 1. Raise worker.maxReplicas or nothing scales.
+#   - Day-to-day behaviour belongs in each Serve deployment's
+#     `autoscaling_config` under serveApplications. RayService applies Serve
+#     config changes to the running cluster in place; any other change here
+#     rolls a new RayCluster (zero-downtime, but a full head and worker
+#     restart); a change to the range alone reaches nothing until the next
+#     roll — patch the RayCluster to apply it live.
+#   - The sidecar reserves 500m CPU / 512Mi memory on the head pod unless
+#     `autoscaling.resources` says otherwise.
+#
+# Full explanation in docs/src/content/docs/scaling.md ("Ray autoscaling").
+# KubeRay background:
+# https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-autoscaling.html
+autoscaling:
+  enabled: false
+
+  # Seconds an idle worker pod is kept before scale-down. 60 is Ray's
+  # general-purpose default; inference workloads pay an image pull and a
+  # model load per new worker, so 300-600 usually churns less.
+  idleTimeoutSeconds: 60
+
+  # upscalingMode — read by the Ray autoscaler, not the KubeRay operator:
+  #   - `Default`      — upscaling is not rate-limited
+  #   - `Conservative` — rate-limited: pending worker pods are capped at the
+  #                      number of workers already connected to the cluster
+  #   - `Aggressive`   — an alias for `Default`; behaviour is identical
+  upscalingMode: Default
+
+  # Autoscaler sidecar resources. Left unset ({}), KubeRay applies requests
+  # AND limits of 500m CPU / 512Mi memory:
+  #   resources:
+  #     limits:
+  #       cpu: "500m"
+  #       memory: "512Mi"
+  #     requests:
+  #       cpu: "500m"
+  #       memory: "512Mi"
+  resources: {}
+
 # =============================================================================
 # Overrides
 # =============================================================================

docs/src/content/docs/configuration.md

@@ -34,8 +34,8 @@ see [Connecting from Jupyter](/jupyter/#versions-must-match).
 | Value | Default | Purpose |
 |---|---|---|
 | `worker.replicas` | `1` | Worker pods. |
-| `worker.minReplicas` | `1` | Lower clamp on `replicas`. Pinned to `1` in `values.yaml` — it does not follow `replicas`. |
-| `worker.maxReplicas` | `1` | Upper clamp on `replicas`. Same caveat; leave it below `replicas` and you get `maxReplicas` workers. |
+| `worker.minReplicas` | `1` | Lower clamp on `replicas`; the autoscaler's floor under [`autoscaling`](#autoscaling). Pinned to `1` in `values.yaml` — it does not follow `replicas`. |
+| `worker.maxReplicas` | `1` | Upper clamp on `replicas`; the autoscaler's ceiling under [`autoscaling`](#autoscaling). Same caveat; leave it below `replicas` and you get `maxReplicas` workers. |
 | `worker.resources.requests` | `cpu: 1`, `memory: 2Gi` | — |
 | `worker.resources.limits` | `cpu: 2`, `memory: 4Gi` | — |
 | `worker.runtimeClassName` | unset | e.g. `nvidia`. |
@@ -50,10 +50,29 @@ fall back to KubeRay's default. Full rationale in [Scaling and GPUs](/scaling/#p
 :::
 
 :::caution[Always set `replicas`, `minReplicas`, and `maxReplicas` together]
-The chart does not enable `enableInTreeAutoscaling`, so there is no Ray autoscaler and the
-group size is exactly `replicas` — clamped into `[minReplicas, maxReplicas]` by KubeRay.
-Since `values.yaml` pins both bounds to `1`, raising `replicas` alone changes nothing. See
+With `autoscaling.enabled: false` (the default) there is no Ray autoscaler, and the group
+size is exactly `replicas` — clamped into `[minReplicas, maxReplicas]` by KubeRay. Since
+`values.yaml` pins both bounds to `1`, raising `replicas` alone changes nothing. See
 [Scaling and GPUs](/scaling/#adding-workers).
+
+With autoscaling on, `replicas` is only the starting size and the bounds are the range the
+autoscaler works within — still `1..1` by default, so raise `maxReplicas`.
+:::
+
+## `autoscaling`
+
+| Value | Default | Purpose |
+|---|---|---|
+| `autoscaling.enabled` | `false` | Emits `enableInTreeAutoscaling: true`, so KubeRay attaches a Ray autoscaler sidecar to the head pod. |
+| `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down. |
+| `autoscaling.upscalingMode` | `Default` | `Default` (not rate-limited), `Conservative` (pending pods capped at connected workers), or `Aggressive` (an alias for `Default`). |
+| `autoscaling.resources` | `{}` | Overrides the sidecar's resources. Unset, KubeRay hardcodes requests **and** limits of `500m` CPU / `512Mi` memory. |
+
+:::note[The range is a budget, not a knob]
+Raise `worker.maxReplicas` — both bounds default to `1`. Then leave the range alone: a
+values-only change to it does not reach a running cluster at all, and changing any other
+`autoscaling.*` value rolls a new RayCluster, whereas a Serve deployment's own
+`autoscaling_config` applies in place. See [Ray autoscaling](/scaling/#ray-autoscaling).
 :::
 
 ## `serve` and `serveApplications`

docs/src/content/docs/scaling.md

@@ -41,15 +41,139 @@ worker:
 replicas into the `[minReplicas, maxReplicas]` range.
 :::
 
-There is no autoscaler behind these bounds. The chart does not set
-`enableInTreeAutoscaling`, so KubeRay runs no Ray autoscaler sidecar and the group size is
-exactly `replicas`. Growing the pool means changing `replicas` and running `helm upgrade`;
-`minReplicas` and `maxReplicas` are clamps, not a range something moves within.
+By default there is no autoscaler behind these bounds: the group size is exactly `replicas`,
+and growing the pool means changing `replicas` and running `helm upgrade`. `minReplicas` and
+`maxReplicas` only become a range something moves within under
+[Ray autoscaling](#ray-autoscaling).
+
+:::caution[Changing the worker count does not reach a running cluster]
+A `helm upgrade` that changes only `replicas`, `minReplicas`, or `maxReplicas` updates the
+RayService and stops there. The RayService controller leaves those three fields out of the
+spec hash it uses to detect drift, so the running RayCluster keeps its old numbers — no event,
+no rollout, no error. The change lands only when some other `rayClusterConfig` field changes in
+the same upgrade, and then as a full cluster roll. To add workers to a running cluster, patch
+the RayCluster directly and put the same numbers in your values file so the next rollout
+carries them:
+
+```bash
+kubectl -n rayserve patch raycluster <name> --type json -p '[
+  {"op":"replace","path":"/spec/workerGroupSpecs/0/replicas","value":6},
+  {"op":"replace","path":"/spec/workerGroupSpecs/0/minReplicas","value":6},
+  {"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":6}]'
+```
+
+KubeRay adds the pods in place. See
+[why a range change does not reach the cluster](#why-a-range-change-does-not-reach-the-cluster);
+a chart-side fix is tracked in [#41](https://github.com/nebari-dev/rayserve-pack/issues/41).
+:::
 
 Whether the new pods actually land is a separate question. On a cluster with a node
 autoscaler, asking for more than current nodes can hold triggers node scale-up; without one,
 the extra pods stay `Pending`.
 
+## Ray autoscaling
+
+```yaml
+autoscaling:
+  enabled: true
+
+worker:
+  minReplicas: 1
+  maxReplicas: 6
+  resources:
+    requests: { cpu: "4", memory: "16Gi" }
+    limits:   { cpu: "4", memory: "16Gi" }
+
+serveApplications:
+  - name: my-model
+    import_path: myapp.model:app
+    deployments:
+      - name: MyModel
+        ray_actor_options: { num_cpus: 4 }
+        autoscaling_config:
+          min_replicas: 1
+          max_replicas: 6
+          target_ongoing_requests: 2
+```
+
+Three layers, each reacting to the one above it:
+
+1. **Serve deployment autoscaling** (`autoscaling_config`) adds model replicas as request load
+   rises. Each replica needs the resources declared in `ray_actor_options`.
+2. **The Ray autoscaler**, which `autoscaling.enabled` turns on, adds worker pods when those
+   replicas have nowhere to run, within `[worker.minReplicas, worker.maxReplicas]`, and removes
+   pods that have been idle for `autoscaling.idleTimeoutSeconds`.
+3. **Your node autoscaler** adds nodes when the pods cannot be scheduled.
+
+The Ray autoscaler reacts to any unschedulable demand, so on a cluster that notebooks connect
+to over Ray client, users' tasks and actors are the first layer as much as Serve is. For a
+Serve-only cluster, a fixed `num_replicas` never generates demand and the autoscaler has
+nothing to do; and without the second layer, `autoscaling_config` can only scale within the
+resources the cluster already has.
+
+:::caution[Raise `maxReplicas`]
+Both bounds default to `1`. `autoscaling.enabled: true` alone leaves the group pinned at one
+worker and only adds the autoscaler sidecar to the head pod — `500m` CPU / `512Mi` memory,
+requests and limits, unless `autoscaling.resources` overrides it.
+:::
+
+### Tune Serve, set the cluster range once
+
+The two layers differ in how a change reaches a running service:
+
+- **Serve config** — `serveApplications`, including `autoscaling_config` — is applied in
+  place. The RayService controller resubmits it to the live cluster and replicas adjust in
+  seconds.
+- **Cluster config** — everything under `rayClusterConfig`, which includes `autoscaling.*` —
+  is replaced, not edited. Any change rolls a new RayCluster: a second head and worker set
+  start, Serve comes up on them, traffic switches, the old cluster is deleted. Zero downtime,
+  but a full model reload. The worker range is the exception: on its own it changes nothing
+  until something else rolls the cluster.
+
+So treat `worker.minReplicas` / `worker.maxReplicas` as a capacity budget — the most workers
+you will pay for, the fewest you want warm — set at install and rarely revisited. Put the
+behaviour you expect to tune in `autoscaling_config`: `min_replicas` / `max_replicas`,
+`target_ongoing_requests`, `upscale_delay_s` / `downscale_delay_s`.
+
+If you need a new ceiling on a live service, patch the RayCluster directly. The controller
+leaves the range fields to the autoscaler and will not revert it; put the same value in your
+values file so the next rollout carries it:
+
+```bash
+kubectl -n rayserve patch raycluster <name> --type json \
+  -p '[{"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":8}]'
+```
+
+### Why a range change does not reach the cluster
+
+The RayService controller excludes `replicas`, `minReplicas`, and `maxReplicas` from the spec
+hash it uses to detect drift, because the autoscaler writes `replicas` itself. So a
+`helm upgrade` changing only those fields — the autoscaling range, or the static worker
+count — never reaches the running cluster. Upstream considers this intended
+([kuberay #2331](https://github.com/ray-project/kuberay/issues/2331)); a fix that propagated
+the range in place was declined
+([kuberay #2333](https://github.com/ray-project/kuberay/pull/2333)), with the guidance being to
+edit the RayCluster directly, as above. A chart hook that applies the values to the live
+RayCluster automatically is proposed in
+[#41](https://github.com/nebari-dev/rayserve-pack/issues/41).
+
+### Idle timeout and upscaling mode
+
+`idleTimeoutSeconds: 60` is Ray's general-purpose default. A new worker pays an image pull and
+a model load, so for inference a longer hold — 300 to 600 seconds — usually beats reclaiming a
+pod that will be wanted again two minutes later.
+
+`upscalingMode: Default` is right for Serve. `Conservative` caps pending worker pods at the
+number already connected, which only helps when node provisioning is slow and a burst of
+`Pending` pods causes trouble. `Aggressive` is an alias for `Default`.
+
+### With Argo CD
+
+`enableInTreeAutoscaling` renders under `spec.rayClusterConfig` — the path the documented
+Application ignores under `RespectIgnoreDifferences=true`. Enabling autoscaling on an
+already-synced cluster is silently not applied unless that ignore rule is narrowed. Same issue
+as [`orgCABundle`](/ca-bundle/).
+
 ## GPUs
 
 Four things have to line up — and the fourth is the one people miss.
@@ -206,12 +330,9 @@ head that starts OOM-killing takes the whole cluster with it, so it is worth hea
 
 ## What is not here
 
-- **Ray cluster autoscaling** — the chart does not set `enableInTreeAutoscaling`, so the
-  worker group never grows on its own.
 - **Per-deployment autoscaling** — Ray Serve's own `autoscaling_config` goes in a
-  `serveApplications` deployment entry, not in the chart's values. It scales replicas within
-  the resources the cluster already has, which is all it can do without the cluster
-  autoscaler above.
+  `serveApplications` deployment entry, not in chart values. [Ray autoscaling](#ray-autoscaling)
+  covers how it pairs with the cluster autoscaler.
 - **Multiple worker groups** — the chart renders one `workerGroupSpecs` entry. Heterogeneous
   pools (CPU plus GPU) need a chart change or a second release.
 - **Node autoscaling** — that is your cluster autoscaler's job.