Skip to content

feat(smoke-test): mount qwen3-0.6b base model via weights API instead of HF - #118

Open
William-Gao1 wants to merge 2 commits into
mainfrom
will/smoke-test-bdn-weights
Open

feat(smoke-test): mount qwen3-0.6b base model via weights API instead of HF#118
William-Gao1 wants to merge 2 commits into
mainfrom
will/smoke-test-bdn-weights

Conversation

@William-Gao1

@William-Gao1 William-Gao1 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

This PR makes two changes to unblock and harden the training smoke test.

1. Mount qwen3-0.6b base model via the weights API (BDN)

The smoke test pulled Qwen/Qwen3-0.6B directly from HuggingFace inside the training container on every run (via axolotl resolving base_model). With the daily schedule plus the H100/H200 matrix, this was getting rate-limited by HF.

  • config.py — add WeightsSource(source="hf://Qwen/Qwen3-0.6B", mount_location="/mnt/user/Qwen3-0.6B") to the TrainingJob, and pass the mount path to the container via the MODEL_MOUNT_PATH env var. Baseten mirrors the repo into BDN once (deduplicated at the WeightVersion level) and CSI-mounts it.

  • train.pyseed_hf_cache_from_mount() wires the mounted weights into the HF hub cache under the canonical repo id (real commit from a cheap metadata call → snapshots/<commit> symlinked to the mount). from_pretrained("Qwen/Qwen3-0.6B") then loads from the mount (only lightweight HEADs, no 1.5 GB download) and records the HF id in the saved LoRA adapter, so the checkpoint is deployable — deploy resolves the base from the BDN mirror by namespace/model id.

    (Recording the id at train time is deliberate: a post-hoc rewrite of adapter_config.json loses a race — the job is COMPLETED and deployed before the re-upload + ensure_checkpoints re-read propagate the new base_model.)

2. Fix deploy-and-infer: pass run_id

truss's create_model_version_from_inference_template() gained a required run_id positional arg, so the deploy step failed with missing 1 required positional argument: 'run_id' even after training completed and checkpointed. The smoke test deploys training-job checkpoints, and _hydrate_deploy_config raises if run_id is set alongside training-job checkpoints, so the correct value is None.

Notes

  • First run still seeds the BDN mirror from HF once; the win is on repeated runs. The dataset (winglian/pirate-ultrachat-10k) is still an HF pull.
  • Separately, there's an intermittent dataset-prep failure (FileNotFoundError on an arrow shard under the project cache mount during axolotl's 128-process tokenization) that predates this PR and is not addressed here.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown

Example Validation Results

examples/qwen3-0.6b-axolotl/training/config.py

Status: TRAINING_JOB_COMPLETED | Job: q8o1yy3 | Project: pr-118-examples--qwen3-0.6b-axolotl

Cache Summary
Error fetching cache summary: 404 Client Error: Not Found for url: https://api.baseten.co/v1/training_projects/6wg6gg3/cache/summary
Client error: No cache summary data found for this project. Ensure you're running jobs with cache enabled.
ERROR HTTPError: 404 Client Error: Not Found for url: 
https://api.baseten.co/v1/training_projects/6wg6gg3/cache/summary
Checkpoints
2 checkpoint(s)

ID                   Type       Size         Created
-------------------- ---------- ------------ -------------------
.                    lora       996.93 MB    2026-06-30 16:32:18
checkpoint-10        lora       981.05 MB    2026-06-30 16:33:31

@William-Gao1
William-Gao1 force-pushed the will/smoke-test-bdn-weights branch 3 times, most recently from d91f457 to ae43725 Compare June 30, 2026 16:18
… of HF

The smoke test pulled Qwen/Qwen3-0.6B directly from HuggingFace inside the
training container on every run, which is getting rate-limited by HF. Mount
the base model through the truss-train weights API instead: Baseten mirrors it
into the Baseten Delivery Network once and serves it from cache on subsequent
runs, so repeated CI runs no longer hit HF.

- config.py: add WeightsSource(hf://Qwen/Qwen3-0.6B) mounted at /mnt/user/Qwen3-0.6B,
  and pass the mount path to the container via the BASE_MODEL_PATH env var so it
  stays defined in one place.
- train.py: read BASE_MODEL_PATH and point axolotl base_model at the mounted path.
…emplate

truss's create_model_version_from_inference_template() gained a required
`run_id` positional argument, so the deploy-and-infer step of the smoke test
fails with "missing 1 required positional argument: 'run_id'" even when the
training job completes and checkpoints successfully.

The smoke test deploys training-job checkpoints, and _hydrate_deploy_config
raises if run_id is set alongside training-job checkpoints, so the correct
value here is None.
@William-Gao1
William-Gao1 force-pushed the will/smoke-test-bdn-weights branch from ae43725 to 1c682cc Compare June 30, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant