Skip to content

decouple trainer step completion from weight broadcast#3024

Draft
tim0120 wants to merge 1 commit into
feat/static-sftfrom
agent/trainer-step-ack
Draft

decouple trainer step completion from weight broadcast#3024
tim0120 wants to merge 1 commit into
feat/static-sftfrom
agent/trainer-step-ack

Conversation

@tim0120

@tim0120 tim0120 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • publish a generic per-run trainer-step completion marker after each optimizer update
  • make static SFT wait on that marker instead of a filesystem weight broadcast
  • restrict filesystem and NCCL weight publication to ready runs that actually need inference
  • move ready_to_update release out of broadcaster implementations and into trainer-step completion

This keeps WeightWatcher weight-specific and avoids full-model or adapter serialization for static SFT. Policy-backed RL runs continue to use their existing weight broadcast paths.

Multi-run behavior

A trainer step may update static and policy-backed runs together. Only policy-backed runs are included in weight publication; all updated runs receive their own trainer_steps/step_N/STABLE completion marker and are released for their next batch.

Validation

  • focused trainer state test: 1 passed
  • Ruff on all changed Python files: passed
  • diff whitespace check: passed

The repository lock currently supports Linux only, so the focused test was run on macOS with an ephemeral uv --no-project dependency environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant