Add experimental bucket+mount transport for Jobs script upload#4025
Open
davanstrien wants to merge 4 commits intomainfrom
Open
Add experimental bucket+mount transport for Jobs script upload#4025davanstrien wants to merge 4 commits intomainfrom
davanstrien wants to merge 4 commits intomainfrom
Conversation
When `HF_JOBS_USE_BUCKET_TRANSPORT=1` is set, `run_uv_job` and
`create_scheduled_uv_job` upload local scripts to a
`{namespace}/jobs-artifacts` bucket instead of base64-encoding them
into an environment variable. The bucket is mounted at `/artifacts`
and the job runs scripts directly from disk.
Falls back to base64 transport if bucket creation fails, hf_xet is
unavailable, or the `/artifacts` mount path is already taken by a
user-provided volume. The existing base64 path is unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Member
Author
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
lhoestq
reviewed
Apr 2, 2026
Comment on lines
+11923
to
+11925
| # Bucket transport constants for Jobs | ||
| _HF_JOBS_ARTIFACTS_MOUNT_PATH = "/artifacts" | ||
| _HF_JOBS_ARTIFACTS_BUCKET_NAME = "jobs-artifacts" |
| HF_XET_HIGH_PERFORMANCE: bool = _is_true(os.environ.get("HF_XET_HIGH_PERFORMANCE")) | ||
|
|
||
| # Opt-in to bucket-based script transport for Jobs (experimental) | ||
| HF_JOBS_USE_BUCKET_TRANSPORT: bool = _is_true(os.environ.get("HF_JOBS_USE_BUCKET_TRANSPORT")) |
Member
There was a problem hiding this comment.
imo we can already set it to True by default
Member
Author
There was a problem hiding this comment.
probably makes sense indeed! I'll wait to see what @Wauplin thinks
Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Follows lhoestq's review suggestion in #4025. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This is the first step in broader work to integrate buckets with Jobs, with follow-up PRs to come as the design iterates (cleanup,
--data/--data-outflags, configurable bucket ID, etc.).The current base64 transport works but requires a slightly fragile
bash -c+xargs+base64 -ddecoder pipeline with manual shell quoting. The bucket path produces a plainuv run /artifacts/script.pycommand — simpler and easier to debug. It also lets jobs write output artifacts back to/artifacts/, which base64 can't do.Summary
HF_JOBS_USE_BUCKET_TRANSPORT=1environment variable{namespace}/jobs-artifactsbucket under_scripts/{uuid}/, mounted at/artifactshf_xetis unavailable, or/artifactsis already taken by a user volumeOpen questions
jobs-artifacts.Happy to iterate on the patterns here — this is a draft to get feedback on the direction.
Test plan
/artifacts/Note
Medium Risk
Adds a new opt-in Jobs execution path that creates/uses buckets and mounts volumes, introducing new network/storage side effects and fallback logic that could affect job submission when enabled.
Overview
Adds an experimental, opt-in Jobs script transport controlled by
HF_JOBS_USE_BUCKET_TRANSPORT, uploading localuvjob scripts/configs to a{namespace}/jobs-artifactsbucket and mounting it at/artifactsso jobs run via a plainuv runcommand.Updates
run_uv_job/create_scheduled_uv_jobto accept and merge any returned extra bucketVolumes, and extends_create_uv_command_env_and_secretsto choose between bucket transport and the existing base64bash -ctransport with explicit fallbacks (bucket upload failure, missinghf_xet, or/artifactsmount collision). Adds unit tests covering the new bucket path and fallback scenarios.Written by Cursor Bugbot for commit 93e0070. This will update automatically on new commits. Configure here.