diff --git a/README.md b/README.md index 785a93f..363b561 100644 --- a/README.md +++ b/README.md @@ -214,11 +214,15 @@ Key values in `chart/values.yaml`: | `head.resources.requests.memory` | `2Gi` | Head node memory request | | `head.runtimeClassName` | - | Runtime class for head pod (e.g., `nvidia` for GPU) | | `worker.replicas` | `1` | Number of worker nodes | -| `worker.minReplicas` | `1` | Min workers (for autoscaling) | -| `worker.maxReplicas` | `1` | Max workers (for autoscaling) | +| `worker.minReplicas` | `1` | Lower bound on workers; the autoscaler's floor when `autoscaling.enabled` | +| `worker.maxReplicas` | `1` | Upper bound on workers; the autoscaler's ceiling when `autoscaling.enabled`. Raise it or autoscaling does nothing | | `worker.resources.requests.cpu` | `1` | Worker CPU request | | `worker.resources.requests.memory` | `2Gi` | Worker memory request | | `worker.runtimeClassName` | - | Runtime class for worker pods (e.g., `nvidia` for GPU) | +| `autoscaling.enabled` | `false` | Enable Ray in-tree autoscaling of worker pods | +| `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down | +| `autoscaling.upscalingMode` | `Default` | `Default` or `Conservative` (rate-limited); `Aggressive` is an alias for `Default` | +| `autoscaling.resources` | `{}` | Autoscaler sidecar resources; KubeRay's default is `500m` / `512Mi` for both requests and limits | ### Serve Applications diff --git a/chart/Chart.yaml b/chart/Chart.yaml index ba07c8b..8f0c259 100644 --- a/chart/Chart.yaml +++ b/chart/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: nebari-rayserve-pack description: A Nebari Software Pack for Ray Serve type: application -version: 0.4.1 +version: 0.5.0 appVersion: "2.43.0" dependencies: - name: kuberay-operator diff --git a/chart/templates/NOTES.txt b/chart/templates/NOTES.txt index 4794381..ff26fa3 100644 --- a/chart/templates/NOTES.txt +++ b/chart/templates/NOTES.txt @@ -3,6 +3,15 @@ The RayService controller manages the Ray cluster and Serve proxy. Serve is pre-initialized with host 0.0.0.0 on port 8000. +{{- if .Values.autoscaling.enabled }} +{{- if not (gt (int .Values.worker.maxReplicas) (int .Values.worker.minReplicas)) }} + +WARNING: autoscaling.enabled is true but the worker range is + [{{ .Values.worker.minReplicas }}, {{ .Values.worker.maxReplicas }}], so nothing can scale. Raise worker.maxReplicas. + +{{- end }} +{{- end }} + {{- if .Values.nebariapp.enabled }} NebariApp is ENABLED. The nebari-operator will configure: diff --git a/chart/templates/rayservice.yaml b/chart/templates/rayservice.yaml index b2475ee..a08efdc 100644 --- a/chart/templates/rayservice.yaml +++ b/chart/templates/rayservice.yaml @@ -19,6 +19,16 @@ spec: rayClusterConfig: rayVersion: {{ .Values.image.tag | quote }} + {{- if .Values.autoscaling.enabled }} + enableInTreeAutoscaling: true + autoscalerOptions: + idleTimeoutSeconds: {{ .Values.autoscaling.idleTimeoutSeconds }} + upscalingMode: {{ .Values.autoscaling.upscalingMode | quote }} + {{- with .Values.autoscaling.resources }} + resources: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} headGroupSpec: rayStartParams: dashboard-host: "0.0.0.0" diff --git a/chart/values.yaml b/chart/values.yaml index b25c4d7..6befea8 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -162,6 +162,9 @@ head: # Ray Workers # ============================================================================= worker: + # A values-only change to these does not reach a running RayCluster: KubeRay + # leaves them to the autoscaler. Patch the RayCluster in place instead (see + # docs, "Adding workers"). Chart-side fix tracked in rayserve-pack#41. replicas: 1 minReplicas: 1 maxReplicas: 1 @@ -222,6 +225,55 @@ worker: timeoutSeconds: 2 failureThreshold: 120 +# ============================================================================= +# Ray In-Tree Autoscaling +# ============================================================================= +# Off by default. When enabled, the chart sets `enableInTreeAutoscaling: true` and +# KubeRay attaches the Ray autoscaler as a sidecar on the head pod. It adds +# worker pods within [worker.minReplicas, worker.maxReplicas] when Ray tasks, +# actors, or Serve replicas have nowhere to run, and removes pods idle for +# idleTimeoutSeconds; worker.replicas is only the starting size. +# +# Treat the range as a capacity budget set once, not a tuning knob: +# - Both bounds default to 1. Raise worker.maxReplicas or nothing scales. +# - Day-to-day behaviour belongs in each Serve deployment's +# `autoscaling_config` under serveApplications. RayService applies Serve +# config changes to the running cluster in place; any other change here +# rolls a new RayCluster (zero-downtime, but a full head and worker +# restart); a change to the range alone reaches nothing until the next +# roll — patch the RayCluster to apply it live. +# - The sidecar reserves 500m CPU / 512Mi memory on the head pod unless +# `autoscaling.resources` says otherwise. +# +# Full explanation in docs/src/content/docs/scaling.md ("Ray autoscaling"). +# KubeRay background: +# https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-autoscaling.html +autoscaling: + enabled: false + + # Seconds an idle worker pod is kept before scale-down. 60 is Ray's + # general-purpose default; inference workloads pay an image pull and a + # model load per new worker, so 300-600 usually churns less. + idleTimeoutSeconds: 60 + + # upscalingMode — read by the Ray autoscaler, not the KubeRay operator: + # - `Default` — upscaling is not rate-limited + # - `Conservative` — rate-limited: pending worker pods are capped at the + # number of workers already connected to the cluster + # - `Aggressive` — an alias for `Default`; behaviour is identical + upscalingMode: Default + + # Autoscaler sidecar resources. Left unset ({}), KubeRay applies requests + # AND limits of 500m CPU / 512Mi memory: + # resources: + # limits: + # cpu: "500m" + # memory: "512Mi" + # requests: + # cpu: "500m" + # memory: "512Mi" + resources: {} + # ============================================================================= # Overrides # ============================================================================= diff --git a/docs/src/content/docs/configuration.md b/docs/src/content/docs/configuration.md index e1f637a..976a40c 100644 --- a/docs/src/content/docs/configuration.md +++ b/docs/src/content/docs/configuration.md @@ -34,8 +34,8 @@ see [Connecting from Jupyter](/jupyter/#versions-must-match). | Value | Default | Purpose | |---|---|---| | `worker.replicas` | `1` | Worker pods. | -| `worker.minReplicas` | `1` | Lower clamp on `replicas`. Pinned to `1` in `values.yaml` — it does not follow `replicas`. | -| `worker.maxReplicas` | `1` | Upper clamp on `replicas`. Same caveat; leave it below `replicas` and you get `maxReplicas` workers. | +| `worker.minReplicas` | `1` | Lower clamp on `replicas`; the autoscaler's floor under [`autoscaling`](#autoscaling). Pinned to `1` in `values.yaml` — it does not follow `replicas`. | +| `worker.maxReplicas` | `1` | Upper clamp on `replicas`; the autoscaler's ceiling under [`autoscaling`](#autoscaling). Same caveat; leave it below `replicas` and you get `maxReplicas` workers. | | `worker.resources.requests` | `cpu: 1`, `memory: 2Gi` | — | | `worker.resources.limits` | `cpu: 2`, `memory: 4Gi` | — | | `worker.runtimeClassName` | unset | e.g. `nvidia`. | @@ -50,10 +50,29 @@ fall back to KubeRay's default. Full rationale in [Scaling and GPUs](/scaling/#p ::: :::caution[Always set `replicas`, `minReplicas`, and `maxReplicas` together] -The chart does not enable `enableInTreeAutoscaling`, so there is no Ray autoscaler and the -group size is exactly `replicas` — clamped into `[minReplicas, maxReplicas]` by KubeRay. -Since `values.yaml` pins both bounds to `1`, raising `replicas` alone changes nothing. See +With `autoscaling.enabled: false` (the default) there is no Ray autoscaler, and the group +size is exactly `replicas` — clamped into `[minReplicas, maxReplicas]` by KubeRay. Since +`values.yaml` pins both bounds to `1`, raising `replicas` alone changes nothing. See [Scaling and GPUs](/scaling/#adding-workers). + +With autoscaling on, `replicas` is only the starting size and the bounds are the range the +autoscaler works within — still `1..1` by default, so raise `maxReplicas`. +::: + +## `autoscaling` + +| Value | Default | Purpose | +|---|---|---| +| `autoscaling.enabled` | `false` | Emits `enableInTreeAutoscaling: true`, so KubeRay attaches a Ray autoscaler sidecar to the head pod. | +| `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down. | +| `autoscaling.upscalingMode` | `Default` | `Default` (not rate-limited), `Conservative` (pending pods capped at connected workers), or `Aggressive` (an alias for `Default`). | +| `autoscaling.resources` | `{}` | Overrides the sidecar's resources. Unset, KubeRay hardcodes requests **and** limits of `500m` CPU / `512Mi` memory. | + +:::note[The range is a budget, not a knob] +Raise `worker.maxReplicas` — both bounds default to `1`. Then leave the range alone: a +values-only change to it does not reach a running cluster at all, and changing any other +`autoscaling.*` value rolls a new RayCluster, whereas a Serve deployment's own +`autoscaling_config` applies in place. See [Ray autoscaling](/scaling/#ray-autoscaling). ::: ## `serve` and `serveApplications` diff --git a/docs/src/content/docs/scaling.md b/docs/src/content/docs/scaling.md index 331fe38..917485b 100644 --- a/docs/src/content/docs/scaling.md +++ b/docs/src/content/docs/scaling.md @@ -41,15 +41,139 @@ worker: replicas into the `[minReplicas, maxReplicas]` range. ::: -There is no autoscaler behind these bounds. The chart does not set -`enableInTreeAutoscaling`, so KubeRay runs no Ray autoscaler sidecar and the group size is -exactly `replicas`. Growing the pool means changing `replicas` and running `helm upgrade`; -`minReplicas` and `maxReplicas` are clamps, not a range something moves within. +By default there is no autoscaler behind these bounds: the group size is exactly `replicas`, +and growing the pool means changing `replicas` and running `helm upgrade`. `minReplicas` and +`maxReplicas` only become a range something moves within under +[Ray autoscaling](#ray-autoscaling). + +:::caution[Changing the worker count does not reach a running cluster] +A `helm upgrade` that changes only `replicas`, `minReplicas`, or `maxReplicas` updates the +RayService and stops there. The RayService controller leaves those three fields out of the +spec hash it uses to detect drift, so the running RayCluster keeps its old numbers — no event, +no rollout, no error. The change lands only when some other `rayClusterConfig` field changes in +the same upgrade, and then as a full cluster roll. To add workers to a running cluster, patch +the RayCluster directly and put the same numbers in your values file so the next rollout +carries them: + +```bash +kubectl -n rayserve patch raycluster --type json -p '[ + {"op":"replace","path":"/spec/workerGroupSpecs/0/replicas","value":6}, + {"op":"replace","path":"/spec/workerGroupSpecs/0/minReplicas","value":6}, + {"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":6}]' +``` + +KubeRay adds the pods in place. See +[why a range change does not reach the cluster](#why-a-range-change-does-not-reach-the-cluster); +a chart-side fix is tracked in [#41](https://github.com/nebari-dev/rayserve-pack/issues/41). +::: Whether the new pods actually land is a separate question. On a cluster with a node autoscaler, asking for more than current nodes can hold triggers node scale-up; without one, the extra pods stay `Pending`. +## Ray autoscaling + +```yaml +autoscaling: + enabled: true + +worker: + minReplicas: 1 + maxReplicas: 6 + resources: + requests: { cpu: "4", memory: "16Gi" } + limits: { cpu: "4", memory: "16Gi" } + +serveApplications: + - name: my-model + import_path: myapp.model:app + deployments: + - name: MyModel + ray_actor_options: { num_cpus: 4 } + autoscaling_config: + min_replicas: 1 + max_replicas: 6 + target_ongoing_requests: 2 +``` + +Three layers, each reacting to the one above it: + +1. **Serve deployment autoscaling** (`autoscaling_config`) adds model replicas as request load + rises. Each replica needs the resources declared in `ray_actor_options`. +2. **The Ray autoscaler**, which `autoscaling.enabled` turns on, adds worker pods when those + replicas have nowhere to run, within `[worker.minReplicas, worker.maxReplicas]`, and removes + pods that have been idle for `autoscaling.idleTimeoutSeconds`. +3. **Your node autoscaler** adds nodes when the pods cannot be scheduled. + +The Ray autoscaler reacts to any unschedulable demand, so on a cluster that notebooks connect +to over Ray client, users' tasks and actors are the first layer as much as Serve is. For a +Serve-only cluster, a fixed `num_replicas` never generates demand and the autoscaler has +nothing to do; and without the second layer, `autoscaling_config` can only scale within the +resources the cluster already has. + +:::caution[Raise `maxReplicas`] +Both bounds default to `1`. `autoscaling.enabled: true` alone leaves the group pinned at one +worker and only adds the autoscaler sidecar to the head pod — `500m` CPU / `512Mi` memory, +requests and limits, unless `autoscaling.resources` overrides it. +::: + +### Tune Serve, set the cluster range once + +The two layers differ in how a change reaches a running service: + +- **Serve config** — `serveApplications`, including `autoscaling_config` — is applied in + place. The RayService controller resubmits it to the live cluster and replicas adjust in + seconds. +- **Cluster config** — everything under `rayClusterConfig`, which includes `autoscaling.*` — + is replaced, not edited. Any change rolls a new RayCluster: a second head and worker set + start, Serve comes up on them, traffic switches, the old cluster is deleted. Zero downtime, + but a full model reload. The worker range is the exception: on its own it changes nothing + until something else rolls the cluster. + +So treat `worker.minReplicas` / `worker.maxReplicas` as a capacity budget — the most workers +you will pay for, the fewest you want warm — set at install and rarely revisited. Put the +behaviour you expect to tune in `autoscaling_config`: `min_replicas` / `max_replicas`, +`target_ongoing_requests`, `upscale_delay_s` / `downscale_delay_s`. + +If you need a new ceiling on a live service, patch the RayCluster directly. The controller +leaves the range fields to the autoscaler and will not revert it; put the same value in your +values file so the next rollout carries it: + +```bash +kubectl -n rayserve patch raycluster --type json \ + -p '[{"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":8}]' +``` + +### Why a range change does not reach the cluster + +The RayService controller excludes `replicas`, `minReplicas`, and `maxReplicas` from the spec +hash it uses to detect drift, because the autoscaler writes `replicas` itself. So a +`helm upgrade` changing only those fields — the autoscaling range, or the static worker +count — never reaches the running cluster. Upstream considers this intended +([kuberay #2331](https://github.com/ray-project/kuberay/issues/2331)); a fix that propagated +the range in place was declined +([kuberay #2333](https://github.com/ray-project/kuberay/pull/2333)), with the guidance being to +edit the RayCluster directly, as above. A chart hook that applies the values to the live +RayCluster automatically is proposed in +[#41](https://github.com/nebari-dev/rayserve-pack/issues/41). + +### Idle timeout and upscaling mode + +`idleTimeoutSeconds: 60` is Ray's general-purpose default. A new worker pays an image pull and +a model load, so for inference a longer hold — 300 to 600 seconds — usually beats reclaiming a +pod that will be wanted again two minutes later. + +`upscalingMode: Default` is right for Serve. `Conservative` caps pending worker pods at the +number already connected, which only helps when node provisioning is slow and a burst of +`Pending` pods causes trouble. `Aggressive` is an alias for `Default`. + +### With Argo CD + +`enableInTreeAutoscaling` renders under `spec.rayClusterConfig` — the path the documented +Application ignores under `RespectIgnoreDifferences=true`. Enabling autoscaling on an +already-synced cluster is silently not applied unless that ignore rule is narrowed. Same issue +as [`orgCABundle`](/ca-bundle/). + ## GPUs Four things have to line up — and the fourth is the one people miss. @@ -206,12 +330,9 @@ head that starts OOM-killing takes the whole cluster with it, so it is worth hea ## What is not here -- **Ray cluster autoscaling** — the chart does not set `enableInTreeAutoscaling`, so the - worker group never grows on its own. - **Per-deployment autoscaling** — Ray Serve's own `autoscaling_config` goes in a - `serveApplications` deployment entry, not in the chart's values. It scales replicas within - the resources the cluster already has, which is all it can do without the cluster - autoscaler above. + `serveApplications` deployment entry, not in chart values. [Ray autoscaling](#ray-autoscaling) + covers how it pairs with the cluster autoscaler. - **Multiple worker groups** — the chart renders one `workerGroupSpecs` entry. Heterogeneous pools (CPU plus GPU) need a chart change or a second release. - **Node autoscaling** — that is your cluster autoscaler's job.