Skip to content

[Core][RuntimeEnv] Add archives plugin - #65830

Open
weiquanlee wants to merge 2 commits into
ray-project:masterfrom
weiquanlee:feature/add_archives_plugin
Open

[Core][RuntimeEnv] Add archives plugin#65830
weiquanlee wants to merge 2 commits into
ray-project:masterfrom
weiquanlee:feature/add_archives_plugin

Conversation

@weiquanlee

@weiquanlee weiquanlee commented Sep 1, 2026

Copy link
Copy Markdown

Description

This PR adds an alpha archives runtime environment option for making remote, non-Python resources such as models and configuration files available to Ray tasks and actors without changing the worker's current directory or PYTHONPATH.

The new option accepts either a remote archive URI or a dictionary mapping names to URIs. Supported formats are .zip, .tar.gz, and .tgz.

This PR:

  • Adds an ArchivesPlugin that downloads, unpacks, caches, and cleans up archives on each worker node.
  • Exposes ray.runtime_env.get_archive_paths() for retrieving local unpacked paths from workers.
  • Deduplicates repeated URIs and prevents concurrent setup from downloading the same archive more than once.
  • Tracks URI references by both URI and plugin type, allowing the same URI to be used independently by working_dir and archives.
  • Adds schema validation, API documentation, usage examples, and unit and integration tests.

Related issues

#65831

Additional information

Example:

from ray.runtime_env import get_archive_paths

runtime_env = {
    "archives": {
        "model": "s3://bucket/model.tar.gz",
        "config": "https://example.com/config.zip",
    }
}

archive_paths = get_archive_paths()

The returned directories are node-local shared cache directories and should be treated as read-only.

Tests run against a clean snapshot of commit 9b37c12c98:

- python -m pytest -q python/ray/tests/unit/test_runtime_env_validation.py -k archives (18 passed)
- python -m pytest -q python/ray/tests/test_runtime_env_agent.py::test_reference_table_distinguishes_uri_types (1 passed)
- python -m pytest -q python/ray/tests/test_runtime_env_archives.py (4 passed)

Signed-off-by: weiquanlee <demonzsong@gmail.com>
@weiquanlee
weiquanlee requested review from a team, edoakes and richardliaw as code owners September 1, 2026 07:59

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the archives runtime environment feature in Ray, allowing remote archives to be downloaded and unpacked on worker nodes, with local paths exposed via the new get_archive_paths API. Feedback on the implementation highlights several critical issues: a potential crash in ArchivesPlugin.create when uri is None which would break all runtime environment setups; a FileNotFoundError risk in delete_uri if the local directory is missing; a NameError in the agent due to an undefined UriType annotation; and case-sensitivity in archive extension validation.

Comment thread python/ray/_private/runtime_env/archives.py
Comment thread python/ray/_private/runtime_env/archives.py
Comment thread python/ray/_private/runtime_env/validation.py
Comment thread python/ray/_private/runtime_env/agent/runtime_env_agent.py

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 9b37c12. Configure here.

Comment thread python/ray/_private/runtime_env/archives.py
Signed-off-by: weiquanlee <demonzsong@gmail.com>
@ray-gardener ray-gardener Bot added core Issues that should be addressed in Ray Core community-contribution Contributed by the community labels Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community core Issues that should be addressed in Ray Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants