[core][sandbox] Add an experimental HTTP API service for Ray Sandbox - #65633
[core][sandbox] Add an experimental HTTP API service for Ray Sandbox#65633xyuzh wants to merge 8 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements an experimental REST API service for Ray Sandbox, featuring a FastAPI application, a detached SandboxHost actor for background execution, Pydantic schemas, and a comprehensive test suite. The review feedback provides valuable recommendations to improve robustness and performance: handling dead actors gracefully during idempotent creation, caching the runsc binary to prevent disk space leaks, querying sandbox descriptions concurrently using asyncio.gather, wrapping the TTL watchdog shutdown in a try-finally block to guarantee actor self-destruction, and catching SandboxError during file writes to return a client-friendly HTTP 400 instead of an HTTP 500.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ray-gardener
Bot
added
docs
3569f56 to
da92929
Compare
dstrodtman
left a comment
dstrodtman
left a comment
There was a problem hiding this comment.
Docs-team review from Douglas Strodtman (Anyscale documentation team), style and grammar only. Claude Code assisted with the mechanics; I read every line and stand behind it.
The prose in the new HTTP API service section is in good shape after your last pass: soft-wrapped, no em dashes, the semicolons and the mid-sentence colon split into sentences, and the bullets read cleanly as a definition list. Approving on the docs side.
Two optional nits below, both stray semicolons in the endpoints table that the rest of the table already splits. Not blocking.
Scope note: this approval covers the prose in doc/source/ray-core/sandboxes.md. The Python and the API design belong to ray-project/ray-core, which still owns the code paths here.
| os.chmod(tmp_path, 0o755) | ||
| os.replace(tmp_path, runsc_path) | ||
| if shared_bin not in os.environ.get("PATH", "").split(os.pathsep): | ||
| os.environ["PATH"] = f"{shared_bin}{os.pathsep}{os.environ.get('PATH', '')}" |
There was a problem hiding this comment.
Predictable /tmp runsc is hijackable
Medium Severity
The shared runsc cache now lives at a predictable world-writable path. Another local user can pre-create /tmp/ray-sandbox-runsc/runsc; os.path.exists then skips the download and that binary is prepended to PATH. The previous mkdtemp location was not guessable. auto_install_runsc is opt-in, but when enabled this replaces the isolation runtime with an attacker-controlled executable.
Reviewed by Cursor Bugbot for commit d76d31e. Configure here.
| # The previous actor died (node loss, OOM). Clear it and | ||
| # fall through to create a fresh sandbox under the same | ||
| # name, keeping the token idempotent across failures. | ||
| resolver.kill(existing) |
There was a problem hiding this comment.
Token retry kills live sandbox
Medium Severity
The new client-token recovery treats every sandbox_not_found from describe as a dead actor and kills it before recreating. _actor_call also maps transient ActorUnavailableError to that same code, so a brief node or GCS blip plus an idempotent create retry destroys a still-living sandbox and returns a fresh empty one.
Reviewed by Cursor Bugbot for commit d76d31e. Configure here.
|
|
||
| ## HTTP API service | ||
|
|
||
| Ray Sandbox ships an experimental REST API service so you can manage sandboxes from outside the Ray cluster with nothing but an HTTP client and a bearer token. The service is a FastAPI app on Ray Serve (`ray.experimental.sandbox.http`). Each sandbox is held by a named, detached actor, so the service itself is stateless and its replicas can scale or restart without losing sandboxes. |
There was a problem hiding this comment.
I believe named detached actors are currently not as scalable as normal actors (e.g. each actor creation has to wait for a roundtrip to the GCS to make sure the names are unique), it would be great to check with @edoakes or @MengjinYan about that and what we should do (it it is indeed a problem)
|
|
||
| ## HTTP API service | ||
|
|
||
| Ray Sandbox ships an experimental REST API service so you can manage sandboxes from outside the Ray cluster with nothing but an HTTP client and a bearer token. The service is a FastAPI app on Ray Serve (`ray.experimental.sandbox.http`). Each sandbox is held by a named, detached actor, so the service itself is stateless and its replicas can scale or restart without losing sandboxes. |
There was a problem hiding this comment.
Why make sandboxes detached actors instead of replicas? We can treat sandboxes as multiplex ids in serve to do multiplex aware routing, but we can certainly have each serve replica own multiple sandbox actors that can be suspended resumed etc. i think this will be a more scalable system.
A new -e ray-sandbox environment that talks to a deployed ray.experimental.sandbox.http service (ray-project/ray#65633) over plain HTTPS with a bearer token — no ray dependency, no cluster connectivity, zero new Harbor dependencies (httpx and tenacity are core deps): export RAY_SANDBOX_API_URL=https://<service-url> export RAY_SANDBOX_API_KEY=<service bearer token> harbor run -t <task> -e ray-sandbox Sandboxes run the task's prebuilt image under gVisor on the service's Ray cluster. Creation and execution are submit-and-poll so every HTTP request stays short regardless of image pull or command duration; the client retries narrowly (connection errors and 502/503/504 on idempotent calls only, Modal-style) and creation is idempotent via client_token. The task's network policy is authoritative: no-network maps to the service's none mode and a wider --ek network= override is rejected at construction; mode values themselves are validated by the service.
Exposes ray.experimental.sandbox over a versioned REST API (/api/v1) served by Ray Serve, so sandboxes can be managed from outside the Ray cluster with nothing but an HTTP client and a bearer token — e.g. as an Anyscale service, or by agent-evaluation frameworks like Harbor. Design: - Each sandbox is a named, detached SandboxHost actor; the actors are the registry, so the Serve app is stateless and replicas can scale or restart without losing sandboxes. - Creation and execution are async submit + poll (with optional long-poll wait_seconds <= 30s) because image pulls and agent commands outlive HTTP requests and load-balancer limits. - The TTL reclaims both the sandbox and its hosting actor (the core runtime's TTL is deliberately disabled here so there is one owner). - Capabilities, network modes, DNS, shell, and workdir semantics are the core SandboxConfig's (ray-project#65570); the API validates network against VALID_NETWORK_MODES and defaults capabilities to DOCKER_DEFAULT_CAPABILITIES, patching nothing. - fastapi is only needed by this subpackage (ray[serve]); the base sandbox package never imports it. Testing: 54 unit tests run with no cluster and no runsc (fake runtime + fake actor resolver + FastAPI TestClient), including an OpenAPI contract snapshot; a runsc-gated integration test covers the real path. Validated end to end as a local 'serve run' and as an Anyscale service, driving real gVisor sandboxes. Signed-off-by: xyuzh <xinyzng@gmail.com>
POST /sandboxes now synthesizes its 202 response from the request instead of awaiting describe() on the new actor: on a saturated cluster the actor may be queued behind capacity for longer than a client (or load balancer) read timeout, and the endpoint has everything it needs to answer without touching the actor. Surfaced by a Harbor concurrency test that oversubscribed a single node. Signed-off-by: xyuzh <xinyzng@gmail.com>
A 16-way Terminal-Bench run against a cold cluster surfaced two capacity bugs: - The Serve deployment used the default max_ongoing_requests, but this API is long-poll based (requests deliberately hold a slot for up to ~30s), so a handful of concurrent clients saturated the replica and the platform load balancer answered 503 for everyone else. Raise it to 1000; the app is entirely async I/O. - Calls to a SandboxHost whose detached actor exists but has not been *scheduled* yet (cluster autoscaling) block indefinitely. Bound every actor call by the request's own long-poll budget plus a configurable scheduling grace: describe paths report a synthesized 'pending' instead of hanging, and exec/file paths return 409 with a retry hint. Signed-off-by: xyuzh <xinyzng@gmail.com>
An actor whose cpu/memory shape can never fit the cluster raises ActorUnschedulableError from any call; the app let it escape as an opaque 500. Terminal-Bench tasks declaring cpus=4/memory_mb=8192 on a cluster of 4CPU-16GB workers hit this for every large task. Map it to 409 'unschedulable' carrying Ray's own message, so clients see exactly which resource shape cannot be satisfied. Signed-off-by: xyuzh <xinyzng@gmail.com>
Proxies in front of a deployed service cap request bodies (an Anyscale ingress rejected a 4.8MB upload with 413 and killed an 11MB one mid-body), so single-request file uploads have a hidden size ceiling. PUT /files gains an append flag, plumbed through host, runtime, and backend (cat >> instead of cat >), so clients can chunk arbitrarily large uploads into proxy-sized pieces. Signed-off-by: xyuzh <xinyzng@gmail.com>
Match the style established in ray-project#65627 for the networking section: soft-wrapped prose, em-dash and semicolon clauses split into separate sentences, and bold list leads. Also document the new append flag on PUT /files and the 409 unschedulable error code. Signed-off-by: xyuzh <xinyzng@gmail.com>
… shared runsc cache, concurrent listing Three review findings: - A dead detached actor made its client_token permanently return 404; the idempotent-create path now clears the dead actor and creates a fresh sandbox under the same name. - The opt-in runsc download leaked a ~40MB temp dir per boot and raced concurrent boots; it now uses one shared cached path per node with an atomic rename. - GET /sandboxes described actors sequentially; it now gathers concurrently. Signed-off-by: xyuzh <xinyzng@gmail.com>
runsc exec takes numeric -user uid[:gid]; names are resolved against the image's own /etc/passwd host-side. Plumbed through runtime, the HTTP API (StartExecRequest.user), and the backend, with tests. Brings exec to parity with container engines' exec --user and the Harbor environment contract's user= parameter. Signed-off-by: xyuzh <xinyzng@gmail.com>
9ea01b9 to
2af9b07
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Reviewed by Cursor Bugbot for commit 2af9b07. Configure here.
| request.ttl_seconds | ||
| if request.ttl_seconds is not None | ||
| else settings.max_ttl_seconds | ||
| ) |
There was a problem hiding this comment.
Default TTL rejected below one hour
Medium Severity
CreateSandboxRequest.ttl_seconds defaults to 3600, and the handler rejects any value above max_ttl_seconds. A server configured with a max below one hour therefore returns 400 for ordinary creates that omit ttl_seconds. Sending null instead applies the server max, so omitted and explicit-null TTLs disagree.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2af9b07. Configure here.
4 participants
Why are these changes needed?
Ray Sandbox (
ray.experimental.sandbox) can currently only be driven from inside the cluster as Ray actors. This PR adds an experimental HTTP API service so sandboxes can be managed from outside the cluster with nothing but an HTTP client and a bearer token — for example as an Anyscale service, or from agent-evaluation frameworks (a Harborray-sandboxenvironment built on this API is submitted alongside: harbor-framework/harbor PR to follow).This is the second step of the plan from #65570 (first: make the Python API run Docker-built images out of the box; second: make Ray Sandbox consumable as a standalone service).
Design
ray.util.list_named_actors. The service keeps no state, so replicas scale/restart freely.POST /sandboxesandPOST .../execsreturn202immediately; clients poll, optionally long-polling withwait_seconds(≤ 30 s per request). Boot progress is observable:pending → pulling → starting → running | error.SandboxConfig's ([core][sandbox] Make Ray Sandbox run Docker-built images out of the box #65570): capabilities (defaultDOCKER_DEFAULT_CAPABILITIES, sets written exactly), network modes (validated againstVALID_NETWORK_MODES, so a new mode needs no API change),dns,shell(sandbox-level and per-exec override), and the workdir/readonly writability contract. The API patches nothing.Authorization: Bearer <token>enforced whenRAY_SANDBOX_API_TOKENis set; an Anyscale service can leave it unset because the platform edge already requires the service bearer token.GET /healthis always open.ray.experimental.sandbox.httprequiresray[serve]; the base sandbox package never imports it.Endpoints (
/api/v1): create/list/get/delete sandboxes, start/poll execs, PUT/GET files, JSON error envelope (401/404/409/400/413/422/500). Full table in the new docs section.E2E validation
Deployed as an Anyscale service (runsc-equipped cluster image) and as a local
serve run; drove real gVisor sandboxes through create → exec → file transfer → delete, including a full Harbor evaluation run (oracle agent, reward 1.0) against the service URL + token.Related issue number
Follow-up to #65570.
Checks
ruff,pydoclint, docstyle, semgrep, import order) on the changed files.TestClient), including an OpenAPI contract snapshot that pins the v1 surface;test_http_integration.pyis gated on the sandbox test env +runsclike the existing sandbox tests.BUILD.bazeluses the samesandbox_tests/team:coretags so the existing CI job picks the package up.