Skip to content

Stop inverse-square-root cooldowns at zero - #220

Merged
jpuigcerver merged 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix/stop-rsqrt-cooldown-at-zero
Oct 15, 2026
Merged

jpuigcerver merged 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix/stop-rsqrt-cooldown-at-zero

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Summary

Hold the inverse-square-root schedule at zero at and after the end of an enabled cooldown. The current linear expression continues below zero: with peak_value=0.1, decay_steps=12, warmup_steps=4, cooldown_steps=3, and timescale=2, step 13 produces approximately -0.0178174. An optimizer that continues past this boundary resumes parameter updates with a reversed learning-rate sign.

Use an explicit endpoint selection so the result is exactly zero in eager and JIT execution. Preserve the existing warmup, inverse-square-root phase, and in-range cooldown expression. Schedules without a cooldown continue their inverse-square-root tail as before.

Validation

Python 3.12.11, JAX 0.11.2, Flax 0.12.9, Optax 0.2.8, NumPy 2.5.3, macOS arm64 CPU:

  • Against unchanged 6dcc8c0: ten new regression cases fail and two controls pass. All 12 pass with the fix.
  • Tests cover warmup/no warmup, one-/three-step cooldowns, fractional/integer endpoints, eager/JIT evaluation, disabled-cooldown controls, and complete SGD updates with momentum and weight decay beyond the endpoint.
  • New regressions plus optimizer, schedule, training-state, and utility suites: 50 tests pass.
  • All 12 new regressions also pass with JAX_ENABLE_X64=1.
  • Ruff E9/F, selected Pylint error checks, two-space Google YAPF formatting on new tests, compilation, and git diff --check pass.

The full data/checkpoint/training suite, end-to-end image training, and accelerator/multihost execution were not run. This branch changes different files and is independent of the pending weight-decay PR #219.

@sylvesterkaczmarek
sylvesterkaczmarek marked this pull request as ready for review September 22, 2026 22:33
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Could a maintainer review the learning-rate cooldown endpoint fix and unblock the workflow approval and Copybara import?

@jpuigcerver
jpuigcerver merged commit 1b6527a into google-research:main Oct 15, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants