Skip to content

Match weight decay parameters by key - #219

Merged
jpuigcerver merged 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix/weight-decay-parameter-keys
Oct 15, 2026
Merged

jpuigcerver merged 1 commit into
google-research:mainfrom
sylvesterkaczmarek:fix/weight-decay-parameter-keys

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Summary

Look up each flattened parameter by its gradient's key when adding weight decay. Zipping two dictionaries' values pairs unrelated parameters when their insertion orders differ, which occurs naturally when jax.grad rebuilds the gradient tree in key order.

For params = {'b': [10, 20], 'a': [1, 2]} and gradients of the squared norm, decay 0.1 currently produces a=[3, 6], b=[20.1, 40.2] in eager execution instead of a=[2.1, 4.2], b=[21, 42]. Differently shaped leaves can raise a broadcasting error. The jitted path already produces the correct result for this example.

The change matches names explicitly without changing the decay formula, first-match regex rules, or serialization/container handling.

Validation

Python 3.12.11, JAX 0.11.2, Flax 0.12.9, Optax 0.2.8, NumPy 2.5.3, macOS arm64 CPU:

  • New regression module: nine failed and one passed on unchanged 6dcc8c0; all ten pass after the fix.
  • Tests cover nested dictionaries and FrozenDicts, equal/different leaf shapes, uniform/regex decay, eager/JIT parity, and three full SGD updates against a keywise reference.
  • Optimizer, schedule, train-state, utility, and new regression suites: 48 passed.
  • Ruff E9/F, Pylint error checks, YAPF on new tests with two-space Google formatting, Python compilation, and git diff --check pass.

The complete data/checkpoint/training test suite, end-to-end image training, and accelerator/multihost execution were not run. No build metadata or CI configuration is changed.

@sylvesterkaczmarek
sylvesterkaczmarek marked this pull request as ready for review September 22, 2026 21:30
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Could a maintainer review the key-based weight-decay matching and unblock the workflow approval and Copybara import?

@jpuigcerver
jpuigcerver merged commit 3cf7be3 into google-research:main Oct 15, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants