[core][sandbox] Isolate network="public" sandboxes in per-sandbox netns via pasta by xyuzh · Pull Request #65820 · ray-project/ray

xyuzh · 2026-08-31T23:01:53Z

Description

network="public" sandboxes currently run with runsc --network=host in the Ray worker's own network namespace: every sandbox on a node shares one port space, so concurrent workloads that bind a fixed port collide and can reach each other's listeners. The concrete failure is terminal-bench's QEMU tasks (qemu-startup, qemu-alpine-ssh), which start QEMU with hostfwd=tcp::2222-:22 and then SSH to localhost:2222 from inside the same sandbox — under co-tenancy the second bind gets EADDRINUSE, and a verifier can connect to a different sandbox's guest.

This PR gives each public sandbox a private user+network namespace pair bridged by pasta (passt) user-mode networking, the rootless-Podman topology:

a tiny holder process (unshare --user --map-root-user --net) pins the namespaces for the sandbox's lifetime;
pasta attaches from the pod side (--netns/--userns /proc/$PID/ns/*), configures a tap with the pod's addressing, and daemonizes; -t/-u/-T/-U none --no-map-gw make it egress-only — in-sandbox binds are never republished on the pod, pod-local services are unreachable from the sandbox loopback, and there is no inbound path;
runsc run executes inside via nsenter as mapped root (no --rootless; nesting a second userns breaks the gofer's /proc magic-link derefs). runsc still gets --network=host, but "host" is now private to the sandbox. Mount and pid namespaces stay shared, so the bundle and control sockets under --root keep working for pod-side state/exec/kill/delete.

Semantics: public finally matches its documented contract (egress without the host's network identity); host remains the explicit shared-namespace mode. RAY_SANDBOX_PUBLIC_HOST_NETNS=1 on workers restores the previous behavior without a code deploy. The HTTP API gains an opt-in auto_install_pasta setting mirroring auto_install_runsc, and boot best-effort-creates /dev/net/tun (K8s containers ship a minimal /dev without it). Requires pasta and nsenter on nodes for public sandboxes; docs updated (mode table, requirements, install snippets, troubleshooting).

Stacked on #65633 — the first 8 commits are that PR; please review the top 2 commits only until it merges.

Related issues

Related to #65633.

Additional information

Tested with TEST_SANDBOX=1 gated tests in a privileged dev container (non-root user, matching production posture): two concurrent public sandboxes both bind 0.0.0.0:2222 and each reaches its own listener on 127.0.0.1:2222; the worker namespace shows nothing on 2222; no address names one sandbox from another (pasta gives every sandbox the pod's own IP, so cross-sandbox fetches resolve to self or fail); egress + generated-resolv.conf DNS work; teardown reaps the holder/pasta/runsc group with no leaked processes, including the create-failure path. The exact pasta flag list is pinned by a unit test since the flags are the isolation property.

gemini-code-assist

Code Review

This pull request introduces an experimental REST API service for Ray Sandbox, implemented as a FastAPI application on Ray Serve, allowing sandboxes to be managed externally. It also adds support for private network namespaces via pasta for the network="public" mode, enabling port isolation. Feedback focuses on improving robustness: failing fast during namespace holder startup, correctly parsing username:gid during user resolution, preventing a single broken sandbox from failing the list API, and adding timeouts to network downloads of runsc and pasta binaries.

gemini-code-assist · 2026-08-31T23:04:00Z

+            script = (
+                # The holder pins the namespaces for the sandbox's lifetime;
+                # --kill-child ties it to this script's process group.
+                "unshare --user --map-root-user --net --fork --kill-child "
+                f"bash -c 'echo $$ > {pidfile}; exec sleep infinity' & "
+                f"for i in $(seq 1 100); do [ -s {pidfile} ] && break; sleep 0.1; done; "
+                f"NSPID=$(cat {pidfile}); "
+                # pasta runs from the pod side (its uplink is the pod's real
+                # interface), attaches to the holder's namespaces, and
+                # daemonizes; it exits when the namespaces empty.
+                f"{pasta} --netns /proc/$NSPID/ns/net --userns /proc/$NSPID/ns/user && "
+                f"exec nsenter --preserve-credentials -U -n -t $NSPID -- {runsc}"
+            )


The loop waiting for the pidfile to be written runs up to 100 times (10 seconds) even if the background unshare process fails immediately (e.g., due to permission or seccomp errors). Additionally, if NSPID is empty, the script proceeds with empty paths like /proc//ns/net, leading to cryptic failures. We should capture the background PID ($!), check if it is still alive in the loop to fail fast, and validate that NSPID is not empty before proceeding.

Suggested change

script = (

# The holder pins the namespaces for the sandbox's lifetime;

# --kill-child ties it to this script's process group.

"unshare --user --map-root-user --net --fork --kill-child "

f"bash -c 'echo $$ > {pidfile}; exec sleep infinity' & "

f"for i in $(seq 1 100); do [ -s {pidfile} ] && break; sleep 0.1; done; "

f"NSPID=$(cat {pidfile}); "

# pasta runs from the pod side (its uplink is the pod's real

# interface), attaches to the holder's namespaces, and

# daemonizes; it exits when the namespaces empty.

f"{pasta} --netns /proc/$NSPID/ns/net --userns /proc/$NSPID/ns/user && "

f"exec nsenter --preserve-credentials -U -n -t $NSPID -- {runsc}"

)

script = (

# The holder pins the namespaces for the sandbox's lifetime;

# --kill-child ties it to this script's process group.

"unshare --user --map-root-user --net --fork --kill-child "

f"bash -c 'echo $$ > {pidfile}; exec sleep infinity' & "

"HOLDER_PID=$!; "

f"for i in $(seq 1 100); do [ -s {pidfile} ] && break; kill -0 $HOLDER_PID 2>/dev/null || break; sleep 0.1; done; "

f"NSPID=$(cat {pidfile} 2>/dev/null); "

"[ -z '$NSPID' ] && { echo 'Failed to start namespace holder process' >&2; exit 1; }; "

# pasta runs from the pod side (its uplink is the pod's real

# interface), attaches to the holder's namespaces, and

# daemonizes; it exits when the namespaces empty.

f"{pasta} --netns /proc/$NSPID/ns/net --userns /proc/$NSPID/ns/user && "

f"exec nsenter --preserve-credentials -U -n -t $NSPID -- {runsc}"

)

gemini-code-assist · 2026-08-31T23:04:00Z

+        head = user.split(":", 1)[0]
+        if head.isdigit():
+            return user
+        rootfs = os.path.join(self._image_manager.get_image_dir(image), "rootfs")
+        try:
+            passwd = Path(os.path.join(rootfs, "etc", "passwd")).read_text(
+                encoding="utf-8", errors="replace"
+            )
+        except OSError:
+            passwd = ""
+        for line in passwd.splitlines():
+            parts = line.split(":")
+            if len(parts) >= 4 and parts[0] == user:
+                return f"{parts[2]}:{parts[3]}"


If user is passed as username:gid (e.g., postfix:104), parts[0] == user will fail because parts[0] is just the username. This raises SandboxExecError and prevents running as a specific user name with a specific gid. We should split user into head (username) and specified_gid, match parts[0] == head, and use specified_gid if provided.

parts_user = user.split(':', 1) head = parts_user[0] if head.isdigit(): return user specified_gid = parts_user[1] if len(parts_user) > 1 else None rootfs = os.path.join(self._image_manager.get_image_dir(image), 'rootfs') try: passwd = Path(os.path.join(rootfs, 'etc', 'passwd')).read_text( encoding='utf-8', errors='replace' ) except OSError: passwd = '' for line in passwd.splitlines(): parts = line.split(':') if len(parts) >= 4 and parts[0] == head: gid = specified_gid if specified_gid is not None else parts[3] return f'{parts[2]}:{gid}'

gemini-code-assist · 2026-08-31T23:04:00Z

+        for info in results:
+            if isinstance(info, _ApiError) and info.code == "sandbox_not_found":
+                continue
+            if isinstance(info, BaseException):
+                raise info


If any sandbox fails to describe (e.g., due to being permanently unschedulable or having some other actor/system error), list_sandboxes raises the exception, causing the entire GET /sandboxes request to fail. We should log the error and continue/skip the failed sandbox so that a single broken sandbox doesn't break the list API for all other sandboxes.

Suggested change

for info in results:

if isinstance(info, _ApiError) and info.code == "sandbox_not_found":

continue

if isinstance(info, BaseException):

raise info

for info in results:

if isinstance(info, _ApiError) and info.code == 'sandbox_not_found':

continue

if isinstance(info, BaseException):

logger.error('Failed to describe sandbox: %s', info)

continue

gemini-code-assist · 2026-08-31T23:04:00Z

+        tmp_path = f"{runsc_path}.tmp.{os.getpid()}"
+        urllib.request.urlretrieve(url, tmp_path)


Using urllib.request.urlretrieve without a timeout can cause the thread to block indefinitely if the network connection hangs. We should use urllib.request.urlopen with an explicit timeout parameter and write the file using shutil.copyfileobj.

Suggested change

tmp_path = f"{runsc_path}.tmp.{os.getpid()}"

urllib.request.urlretrieve(url, tmp_path)

tmp_path = f'{runsc_path}.tmp.{os.getpid()}'

with urllib.request.urlopen(url, timeout=60) as response:

with open(tmp_path, 'wb') as f:

shutil.copyfileobj(response, f)

gemini-code-assist · 2026-08-31T23:04:00Z

+        tmp_path = f"{pasta_path}.tmp.{os.getpid()}"
+        urllib.request.urlretrieve(url, tmp_path)


Using urllib.request.urlretrieve without a timeout can cause the thread to block indefinitely if the network connection hangs. We should use urllib.request.urlopen with an explicit timeout parameter and write the file using shutil.copyfileobj.

Suggested change

tmp_path = f"{pasta_path}.tmp.{os.getpid()}"

urllib.request.urlretrieve(url, tmp_path)

tmp_path = f'{pasta_path}.tmp.{os.getpid()}'

with urllib.request.urlopen(url, timeout=60) as response:

with open(tmp_path, 'wb') as f:

shutil.copyfileobj(response, f)

Exposes ray.experimental.sandbox over a versioned REST API (/api/v1) served by Ray Serve, so sandboxes can be managed from outside the Ray cluster with nothing but an HTTP client and a bearer token — e.g. as an Anyscale service, or by agent-evaluation frameworks like Harbor. Design: - Each sandbox is a named, detached SandboxHost actor; the actors are the registry, so the Serve app is stateless and replicas can scale or restart without losing sandboxes. - Creation and execution are async submit + poll (with optional long-poll wait_seconds <= 30s) because image pulls and agent commands outlive HTTP requests and load-balancer limits. - The TTL reclaims both the sandbox and its hosting actor (the core runtime's TTL is deliberately disabled here so there is one owner). - Capabilities, network modes, DNS, shell, and workdir semantics are the core SandboxConfig's (ray-project#65570); the API validates network against VALID_NETWORK_MODES and defaults capabilities to DOCKER_DEFAULT_CAPABILITIES, patching nothing. - fastapi is only needed by this subpackage (ray[serve]); the base sandbox package never imports it. Testing: 54 unit tests run with no cluster and no runsc (fake runtime + fake actor resolver + FastAPI TestClient), including an OpenAPI contract snapshot; a runsc-gated integration test covers the real path. Validated end to end as a local 'serve run' and as an Anyscale service, driving real gVisor sandboxes. Signed-off-by: xyuzh <xinyzng@gmail.com>

POST /sandboxes now synthesizes its 202 response from the request instead of awaiting describe() on the new actor: on a saturated cluster the actor may be queued behind capacity for longer than a client (or load balancer) read timeout, and the endpoint has everything it needs to answer without touching the actor. Surfaced by a Harbor concurrency test that oversubscribed a single node. Signed-off-by: xyuzh <xinyzng@gmail.com>

A 16-way Terminal-Bench run against a cold cluster surfaced two capacity bugs: - The Serve deployment used the default max_ongoing_requests, but this API is long-poll based (requests deliberately hold a slot for up to ~30s), so a handful of concurrent clients saturated the replica and the platform load balancer answered 503 for everyone else. Raise it to 1000; the app is entirely async I/O. - Calls to a SandboxHost whose detached actor exists but has not been *scheduled* yet (cluster autoscaling) block indefinitely. Bound every actor call by the request's own long-poll budget plus a configurable scheduling grace: describe paths report a synthesized 'pending' instead of hanging, and exec/file paths return 409 with a retry hint. Signed-off-by: xyuzh <xinyzng@gmail.com>

An actor whose cpu/memory shape can never fit the cluster raises ActorUnschedulableError from any call; the app let it escape as an opaque 500. Terminal-Bench tasks declaring cpus=4/memory_mb=8192 on a cluster of 4CPU-16GB workers hit this for every large task. Map it to 409 'unschedulable' carrying Ray's own message, so clients see exactly which resource shape cannot be satisfied. Signed-off-by: xyuzh <xinyzng@gmail.com>

Proxies in front of a deployed service cap request bodies (an Anyscale ingress rejected a 4.8MB upload with 413 and killed an 11MB one mid-body), so single-request file uploads have a hidden size ceiling. PUT /files gains an append flag, plumbed through host, runtime, and backend (cat >> instead of cat >), so clients can chunk arbitrarily large uploads into proxy-sized pieces. Signed-off-by: xyuzh <xinyzng@gmail.com>

Match the style established in ray-project#65627 for the networking section: soft-wrapped prose, em-dash and semicolon clauses split into separate sentences, and bold list leads. Also document the new append flag on PUT /files and the 409 unschedulable error code. Signed-off-by: xyuzh <xinyzng@gmail.com>

… shared runsc cache, concurrent listing Three review findings: - A dead detached actor made its client_token permanently return 404; the idempotent-create path now clears the dead actor and creates a fresh sandbox under the same name. - The opt-in runsc download leaked a ~40MB temp dir per boot and raced concurrent boots; it now uses one shared cached path per node with an atomic rename. - GET /sandboxes described actors sequentially; it now gathers concurrently. Signed-off-by: xyuzh <xinyzng@gmail.com>

runsc exec takes numeric -user uid[:gid]; names are resolved against the image's own /etc/passwd host-side. Plumbed through runtime, the HTTP API (StartExecRequest.user), and the backend, with tests. Brings exec to parity with container engines' exec --user and the Harbor environment contract's user= parameter. Signed-off-by: xyuzh <xinyzng@gmail.com>

… via pasta Each public-mode sandbox now runs inside its own network namespace bridged by pasta (passt) user-mode networking: internet egress works, but ports and loopback are private — a bind on 0.0.0.0 cannot collide with the worker pod or other sandboxes, and nothing in the sandbox is reachable from outside it. This fulfills public's documented contract (egress without the host's network identity) and unblocks workloads that bind fixed ports, e.g. QEMU hostfwd tasks in Terminal-Bench. runsc still runs with --network=host; host is simply the private namespace. The mount namespace stays shared so runsc control sockets keep working. RAY_SANDBOX_PUBLIC_HOST_NETNS=1 on workers restores the previous shared-namespace behavior without a deploy; the auto_install_pasta server setting (default off) downloads a static pasta build on nodes that lack the passt package. Signed-off-by: xyuzh <xinyzng@gmail.com>

…ns via pasta Each public-mode sandbox gets a private user+network namespace pair, bridged by pasta (passt) user-mode networking in the rootless-Podman topology: a holder process pins the namespaces, pasta attaches from the pod side (so its uplink is the pod's real interface) and daemonizes, and runsc enters via nsenter as mapped root. Internet egress works, but ports and loopback are private — a bind on 0.0.0.0 cannot collide with the worker pod or other sandboxes, and nothing in the sandbox is reachable from outside it. This fulfills public's documented contract (egress without the host's network identity) and unblocks workloads that bind fixed ports, e.g. QEMU hostfwd tasks in Terminal-Bench. runsc still runs with --network=host — "host" is simply the private namespace — and without --rootless: it is already root in the holder's user namespace, and a nested user namespace would put the gofer's /proc/<pid>/root magic links out of reach. Mount and pid namespaces stay shared, so the bundle and the runsc control sockets keep working for pod-side state/exec/kill/delete. The host boot path creates /dev/net/tun (absent in fresh Kubernetes pods) via passwordless sudo. RAY_SANDBOX_PUBLIC_HOST_NETNS=1 on workers restores the previous shared-namespace behavior without a deploy; the auto_install_pasta server setting (default off) downloads a static pasta build on nodes without the passt package. Signed-off-by: xyuzh <xinyzng@gmail.com>

Cut the duplication and the over-long prose the netns tests picked up: - conftest: fold the runsc and pasta downloads into one _install_on_path helper; the two fixtures were the same twenty lines twice. - Share _public_config() and a _run_argv() argv builder instead of rebuilding a backend and config in every test. - Add a no_host_netns fixture for the repeated kill-switch delenv, and parametrize the non-public network modes so a failure names the mode. - Drop the real_which passthrough in the missing-pasta test: patch gvisor.shutil.which by path and return a stub for everything else. - Lift _pasta_pids and _host_ip to module scope, and trim the docstrings and comments to the claim each test actually makes. No coverage changes: same eight tests, same assertions. Signed-off-by: xyuzh <xinyzng@gmail.com>

…ces are forbidden The pasta path parks each sandbox in an `unshare --user --net` holder and has pasta and a non-rootless runsc re-enter it via `nsenter -U -n`. Some sandboxed CI environments allow a single unprivileged user namespace (enough for the rootless sandbox tests) but deny entering another process's, which only surfaces once the sandbox starts as `Couldn't open user namespace ...: Permission denied`. Probe the namespace entry in `ensure_pasta` so those environments skip rather than fail. Signed-off-by: xyuzh <xinyzng@gmail.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 1238fd1. Configure here.}

cursor · 2026-09-01T06:57:50Z

-            if proc and proc.poll() is None:
-                proc.terminate()
-                try:
-                    proc.communicate(timeout=2)


Timeout can kill a running sandbox

Low Severity

The create wait loop now raises SandboxTimeoutError before it polls runsc state. A sandbox that reaches running just after the deadline is torn down by the exception handler even though it started successfully. Pasta startup makes that window easier to hit against the default 30s create timeout.

^{Reviewed by Cursor Bugbot for commit 1238fd1. Configure here.}

xyuzh requested review from a team and andrewsykim as code owners August 31, 2026 23:01

gemini-code-assist Bot reviewed Aug 31, 2026

View reviewed changes

cursor Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread python/ray/experimental/sandbox/backend/gvisor.py

xyuzh mentioned this pull request Sep 1, 2026

[core][sandbox] Support multi-uid user namespaces for network="public" sandboxes #65823

Open

ray-gardener Bot added docs An issue or change related to documentation core Issues that should be addressed in Ray Core labels Sep 1, 2026

xyuzh added 12 commits August 31, 2026 23:47

xyuzh force-pushed the sandbox-public-netns branch from 8e48d3c to 1238fd1 Compare September 1, 2026 06:49

cursor Bot reviewed Sep 1, 2026

View reviewed changes

Navigation Menu

Uh oh!

[core][sandbox] Isolate network="public" sandboxes in per-sandbox netns via pasta - #65820

[core][sandbox] Isolate network="public" sandboxes in per-sandbox netns via pasta#65820
xyuzh wants to merge 12 commits into
ray-project:masterfrom
xyuzh:sandbox-public-netns

xyuzh commented Aug 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		tmp_path = f"{runsc_path}.tmp.{os.getpid()}"
		urllib.request.urlretrieve(url, tmp_path)

-        tmp_path = f"{runsc_path}.tmp.{os.getpid()}"
-        urllib.request.urlretrieve(url, tmp_path)
+        tmp_path = f'{runsc_path}.tmp.{os.getpid()}'
+        with urllib.request.urlopen(url, timeout=60) as response:
+            with open(tmp_path, 'wb') as f:
+                shutil.copyfileobj(response, f)

		tmp_path = f"{pasta_path}.tmp.{os.getpid()}"
		urllib.request.urlretrieve(url, tmp_path)

Uh oh!

Conversation

xyuzh commented Aug 31, 2026

Description

Related issues

Additional information

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 1, 2026

Choose a reason for hiding this comment

Timeout can kill a running sandbox

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant