[core][sandbox] Support multi-uid user namespaces for network="public" sandboxes

[xyuzh]

Description

Rootless-style sandboxes map the whole container to a single host uid: every file reads as root, chown to any other user fails, and ownership baked into image layers flattens away. Workloads that spread ownership across users break — terminal-bench's mailman task fails postfix's ownership integrity check (postsuper: fatal: scan_dir_push: open directory defer: Permission denied) because the image ships /var/spool/postfix/* owned by uid 101 with 0700/1730 modes, which single-uid extraction flattens to root.

This PR gives network="public" sandboxes a multi-uid user namespace, the rootless-Podman model:

• Holder mapping: the namespace holder now starts unmapped, and container root maps onto the worker's ids plus container 1..count onto the node's /etc/subuid//etc/subgid subordinate range before pasta and runsc join. Probe-based detection (detect_idmap()) verifies a mapping can actually be written at boot — via the setuid newuidmap/newgidmap helpers, or via privileged direct /proc/<pid>/uid_map writes where a build pipeline stripped the setuid bits (shadow's helpers refuse cross-user targets, so sudo-ing them is not an option) — and degrades to the previous single-uid namespace with a warn-once when neither works (e.g. no_new_privs, or the pod itself running under a sandboxed runtime such as gVisor/GKE Sandbox whose kernel only accepts self-maps). RAY_SANDBOX_SINGLE_UID=1 forces the fallback.
• Ownership-preserving image cache: extraction records each layer's non-root owners (whiteout-aware) into an .ownership.json sidecar; the cached tar restores true uid/gid onto its members while the shared extracted rootfs stays worker-owned (single-uid sandboxes are unaffected); multi-uid sandboxes mount a per-node <image>.idmap rootfs variant extracted from that tar inside an ephemeral mapped user namespace — chown-before-chmod so setuid/setgid bits survive, out-of-range ids skipped with a warning. The .extracted marker is versioned (ownership-v2) so legacy caches re-pull once.
• Traversing another user's 0700 directory as root additionally needs CAP_DAC_OVERRIDE — present in Docker's default capability set, absent from the bare runsc spec default; documented in troubleshooting.

Node prerequisites (documented): uidmap package + subordinate id ranges for the worker user; runc-backed pods (an outer gVisor sandbox cannot write child uid maps).

Stacked on #65633 and #65820 — review the top commit only until those merge.

Related issues

Related to #65633, #65820.

Additional information

Tested with TEST_SANDBOX=1 gated tests in a privileged dev container as a non-root user, on both mapping paths (setuid helpers intact, and stripped bits + sudo map-file writes): image-baked ownership (uid=101 0700 spool dir, 02710 setgid dir built into a fixture image) stats correctly inside the sandbox and root traverses it; runtime chown to arbitrary uids materializes host-side at the subordinate ids on workdir binds; a mailman-shaped mini (adduser + chown -R + setgid-directory group inheritance) passes; single-uid network="none" sandboxes sharing the same image cache are unaffected; the netns isolation matrix from #65820 passes unchanged under multi-uid.

[gemini-code-assist]

Code Review

This pull request introduces an experimental REST API service for Ray Sandbox, allowing users to manage sandboxes externally via a FastAPI application on Ray Serve. It also adds support for private network namespaces bridged by pasta (passt) and multi-uid user-namespace mapping to preserve file ownership within sandboxes. The review feedback highlights two critical issues: a missing json import in image_utils.py that will cause a runtime NameError, and a potential unhandled OSError in idmap.py if subprocess.Popen fails outside of the try block.

[gemini-code-assist]

The json module is used in _write_ownership_sidecar (via json.dump) and ensure_idmapped_rootfs (via json.dumps), but it is not imported in this file. This will raise a NameError at runtime when these functions are executed. Please import json at the top of the file.

Suggested change

	import subprocess
	import sys
	import json
	import subprocess
	import sys

[gemini-code-assist]

In _probe_sudo_mapfile, subprocess.Popen is called outside of the try block. If unshare is missing or fails to execute (raising an OSError), the exception will propagate unhandled and crash the detection process instead of gracefully falling back to a single-uid namespace. Moving the Popen call inside the try block and checking if holder is not None in the finally block ensures robust error handling.

Suggested change

	holder = subprocess.Popen(
	["unshare", "--user", "sleep", "5"],
	stdin=subprocess.DEVNULL,
	stdout=subprocess.DEVNULL,
	stderr=subprocess.DEVNULL,
	)
	try:
	wait_for_userns(holder.pid)
	map_ids_into(holder.pid, candidate)
	return sudo_mapfile
	except (OSError, RuntimeError):
	pass
	finally:
	holder.kill()
	holder.communicate()
	holder = None
	try:
	holder = subprocess.Popen(
	["unshare", "--user", "sleep", "5"],
	stdin=subprocess.DEVNULL,
	stdout=subprocess.DEVNULL,
	stderr=subprocess.DEVNULL,
	)
	wait_for_userns(holder.pid)
	map_ids_into(holder.pid, candidate)
	return sudo_mapfile
	except (OSError, RuntimeError):
	pass
	finally:
	if holder is not None:
	holder.kill()
	holder.communicate()

[cursor]

Subgid lookup uses gid not uid

Medium Severity

/etc/subgid is keyed by user name or numeric uid, same as /etc/subuid, but detect_idmap looks it up with egid. When the file is uid-keyed and the worker's uid and gid differ, the range is missed and multi-uid silently falls back to a single-uid namespace. _ensure_idmap uses the same gid key when deciding whether a subgid line already exists.

Additional Locations (1)

• python/ray/experimental/sandbox/http/host.py#L167-L174

 

^{Reviewed by Cursor Bugbot for commit f2f0c1b. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit d74d0a1. Configure here.}

[cursor]

Probe skips tests on any failure

Medium Severity

_public_netns_supported treats every SandboxError from create_sandbox as an unsupported host and ensure_pasta then skips the whole netns suite, always blaming a nested user namespace. Image-pull failures, pasta/tun errors, idmap extract problems, timeouts, and regressions in the pasta boot path all become silent skips, so a broken network="public" create no longer fails CI.

Additional Locations (1)

• python/ray/experimental/sandbox/tests/conftest.py#L100-L105

 

^{Reviewed by Cursor Bugbot for commit d74d0a1. Configure here.}