[core] Unify RuntimeEnv archive validation and support tar.xz by zhjwpku · Pull Request #65742 · ray-project/ray

zhjwpku · 2026-08-26T12:17:08Z

Description

RuntimeEnv archive handling currently has duplicated and inconsistent suffix checks across validation, local uploads, package caching, downloading, and the Jobs API. In particular, GCS URIs bypass some validation, working_dir accepts .whl too early, compound suffixes are not always preserved, and .tar.xz is not supported consistently.

Related issues

Closes #65738

Centralize package format capabilities for working_dir and py_modules. Preserve and unpack tar.xz packages across local, remote, GCS, and Jobs paths. Closes ray-project#65738 Signed-off-by: Junwang Zhao <zhjwpku@gmail.com>

gemini-code-assist

Code Review

This pull request adds support for .tar.xz archives in Ray's runtime_env fields (working_dir and py_modules) and centralizes package extension validation logic in a new runtime_env_package.py module. The reviewer feedback highlights several key improvements: parsing the URI before checking its extension in common.py to handle query parameters correctly, sorting supported extensions by length in get_package_extension to prevent incorrect matching of compound extensions, and passing the original user-provided URI to validate_package_extension to ensure clearer error messages when validation fails.

gemini-code-assist · 2026-08-26T12:19:06Z

+    if not has_package_extension(package_uri, PACKAGE_UPLOAD_EXTENSIONS):
+        formats = ", ".join(PACKAGE_UPLOAD_EXTENSIONS)
+        raise ValueError(
+            f"package_uri ({package_uri}) does not end in a supported format: "
+            f"{formats}"
+        )
    # We need to strip the <protocol>:// prefix to make it possible to pass
    # the package_uri over HTTP.
    protocol, package_name = parse_uri(package_uri)


The current implementation checks package_uri directly using has_package_extension. If the URI contains query parameters or fragments (such as presigned S3/GCS URLs or SAS tokens), has_package_extension will return False because the string does not end with the extension.\n\nSince parse_uri already correctly handles stripping query parameters and fragments to extract the clean package_name, we should parse the URI first and then validate the extension on the parsed package_name.

# We need to strip the <protocol>:// prefix to make it possible to pass\n # the package_uri over HTTP.\n protocol, package_name = parse_uri(package_uri)\n if not has_package_extension(package_name, PACKAGE_UPLOAD_EXTENSIONS):\n formats = ', '.join(PACKAGE_UPLOAD_EXTENSIONS)\n raise ValueError(\n f'package_uri ({package_uri}) does not end in a supported format: '\n f'{formats}'\n )

Addressed together with the validation error fix. This call site now passes the query-free original URI as display_path.

gemini-code-assist · 2026-08-26T12:19:06Z

+    for extension in supported_extensions:
+        if path.endswith(extension):
+            return extension


To make get_package_extension robust against the order of extensions in supported_extensions, we should sort them by length in descending order. This ensures that compound extensions (like .tar.gz) are matched before their single-extension counterparts (like .gz) if both are present in the tuple.

for extension in sorted(supported_extensions, key=len, reverse=True):\n if path.endswith(extension):\n return extension

Done. get_package_extension() now sorts the supported extensions by length in descending order before matching.

gemini-code-assist · 2026-08-26T12:19:06Z

+def validate_package_extension(path: str, field: str) -> None:
+    """Validate a RuntimeEnv package path using the field's capabilities.
+
+    Args:
+        path: Package path to validate.
+        field: RuntimeEnv field whose format capabilities apply.
+
+    Raises:
+        ValueError: If the path does not have an extension supported by the field.
+    """
+    supported_extensions = RUNTIME_ENV_PACKAGE_EXTENSIONS[field]
+    if has_package_extension(path, supported_extensions):
+        return
+
+    formats = ", ".join(supported_extensions)
+    raise ValueError(
+        f"Only {formats} files are supported for {field} URIs; got {path}."
+    )


When validation fails, validate_package_extension raises a ValueError displaying the path argument. However, path is often the internal hashed package name (e.g., gcs_digest.txt) returned by parse_uri, rather than the original URI provided by the user (e.g., gcs://package.txt). This makes the error message confusing for users.\n\nWe can add an optional original_path parameter to validate_package_extension to display the original user-provided URI in the error message when available.

def validate_package_extension(\n path: str, field: str, original_path: Optional[str] = None\n) -> None:\n """Validate a RuntimeEnv package path using the field's capabilities.\n\n Args:\n path: Package path to validate.\n field: RuntimeEnv field whose format capabilities apply.\n original_path: Original user-facing path or URI for error messages.\n\n Raises:\n ValueError: If the path does not have an extension supported by the field.\n """\n supported_extensions = RUNTIME_ENV_PACKAGE_EXTENSIONS[field]\n if has_package_extension(path, supported_extensions):\n return\n\n formats = ', '.join(supported_extensions)\n display_path = original_path or path\n raise ValueError(\n f'Only {formats} files are supported for {field} URIs; got {display_path}.'\n )

Ok, I added an optional display_path to validate_package_extension(). The parsed path is still used for validation, while callers provide the original URI with its query removed for the error message.

gemini-code-assist · 2026-08-26T12:19:06Z

+    if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():
+        validate_package_extension(path, field)


Pass the original uri as original_path to validate_package_extension so that the user-facing error message displays the original URI instead of the internal hashed package name.

if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():\n validate_package_extension(path, field, original_path=uri)

Addressed together with the validation error fix. This call site now passes the query-free original URI as display_path.

gemini-code-assist · 2026-08-26T12:19:06Z

+    if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():
+        validate_package_extension(path, PY_MODULES)


Pass the original URI s as original_path to validate_package_extension so that the user-facing error message displays the original URI instead of the internal hashed package name.

if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():\n validate_package_extension(path, PY_MODULES, original_path=s)

gemini-code-assist · 2026-08-26T12:19:06Z

+        if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():
+            validate_package_extension(path, WORKING_DIR)


Pass the original URI working_dir as original_path to validate_package_extension so that the user-facing error message displays the original URI instead of the internal hashed package name.

if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():\n validate_package_extension(path, WORKING_DIR, original_path=working_dir)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c952912ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 7c95291. Configure here.}

Kunchd · 2026-08-31T20:53:48Z

@edoakes Please take a look if you have time.

Signed-off-by: Junwang Zhao <zhjwpku@gmail.com>

[core] Unify RuntimeEnv archive validation and support tar.xz

7c95291

Centralize package format capabilities for working_dir and py_modules. Preserve and unpack tar.xz packages across local, remote, GCS, and Jobs paths. Closes ray-project#65738 Signed-off-by: Junwang Zhao <zhjwpku@gmail.com>

zhjwpku requested review from a team, MengjinYan and edoakes as code owners August 26, 2026 12:17

gemini-code-assist Bot reviewed Aug 26, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread python/ray/_common/runtime_env_package.py

cursor Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread python/ray/tests/test_runtime_env_working_dir.py Outdated

ray-gardener Bot added core Issues that should be addressed in Ray Core community-contribution Contributed by the community labels Aug 26, 2026

Kunchd assigned edoakes Aug 31, 2026

[core] Address RuntimeEnv archive review feedback

8ec03f6

Signed-off-by: Junwang Zhao <zhjwpku@gmail.com>

weiquanlee mentioned this pull request Sep 1, 2026

[Core][RuntimeEnv] Add archives plugin #65830

Open

		if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():
		validate_package_extension(path, field)

		if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():
		validate_package_extension(path, PY_MODULES)

		if protocol == Protocol.GCS or protocol in Protocol.remote_protocols():
		validate_package_extension(path, WORKING_DIR)

Uh oh!

Conversation

zhjwpku commented Aug 26, 2026

Description

Related issues

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

zhjwpku Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

zhjwpku Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

zhjwpku Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

zhjwpku Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

zhjwpku Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

zhjwpku Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kunchd commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants