Skip to content

Confine the auto-resume durability fallback to auto-resume - #194

Merged
Naeemkh merged 1 commit into
mainfrom
fix/auto-resume-durability-fallback
Sep 2, 2026
Merged

Confine the auto-resume durability fallback to auto-resume#194
Naeemkh merged 1 commit into
mainfrom
fix/auto-resume-durability-fallback

Conversation

@Naeemkh

@Naeemkh Naeemkh commented Sep 2, 2026

Copy link
Copy Markdown
Member

Summary

  • _resolve_dcp_load_dir's durability fallback could never fire. explicit tested whether a path argument was passed, but auto-resume always passes the path it resolved, so every production call took the early return and the fallback below was dead code. Now it tests checkpoint.load_path, i.e. whether the user named a checkpoint.
  • Effect: an auto-resume onto a checkpoint whose DCP shards were never made durable now degrades to the last good step instead of dying in dcp.load with metadata is None. An explicit load_path is still honored as-is and still fails loudly.
  • Reachable on the 9 shipped configs using async_with_pinned_mem.

Testing

  • uv run ruff check kempnerforge/ tests/ passes
  • uv run ruff format --check kempnerforge/ tests/ scripts/ passes
  • uv run pyright kempnerforge/ passes (0 errors)
  • uv run pytest tests/unit/ -v --timeout=60 passes
  • If distributed code changed: uv run torchrun --nproc_per_node=4 -m pytest tests/distributed/ -v
  • If training loop / parallelism / optimizers changed: uv run pytest tests/e2e/ --e2e -v

@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
kempnerforge/checkpoint/manager.py 90.63% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Naeemkh
Naeemkh requested review from amazloumi and mmshad September 2, 2026 16:10
@Naeemkh
Naeemkh merged commit 72ad79e into main Sep 2, 2026
6 checks passed
@Naeemkh
Naeemkh deleted the fix/auto-resume-durability-fallback branch September 2, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants