Repository navigation
[BugFix] Make KLCTCLogits._kldiv a batch-mean loss like its twin - #18390
Open
MincongZhou wants to merge 2 commits into
Open
MincongZhou wants to merge 2 commits into
MincongZhou wants to merge 2 commits into
Conversation
ppocr/losses/distillation_loss.py:1108 reduced the KL term twice:
loss = paddle.sum(paddle.mean(loss, axis=1)) / loss.shape[0]
The other copy of the same helper (ppocr/losses/basic_loss.py:114) reduces
once and says so:
# batch mean loss
loss = paddle.sum(loss) / loss.shape[0]
so the distillation version is smaller by the size of axis 1 - a factor of 8
for a 8-way CTC head, and ~6600x for the 6624-character class axis that
`mode='meanlog'` uses. The batch-mean form is the one that matches the
reference: the helper was introduced by PaddlePaddle#5643 and validated there against
`F.kl_div(..., reduction="batchmean")`, and the shipped default
`mode='mean'` already uses the correct `kl_loss`, so as shipped the non-default
modes disagree with the default by that factor.
Axis 1 is the class axis for `forward_log` and the time axis for
`forward_meanlog`, which is why the extra reduction is accidental rather than
deliberate.
Executed with the real class and PaddlePaddle: the two reductions differ by
exactly the axis-1 size (ratio 8.0), and the result stays 0-d, so both call
sites (distillation_loss.py:1091 and :1149) and DistillCTCLogits.forward are
unaffected.
Adds tests/losses/test_distillation_loss.py, which pins the reduction this PR restores: - `_kldiv` equals the single reduction `sum(term) / term.shape[0]`, and not the double reduction `sum(mean(term, axis=1)) / term.shape[0]`; the two differ by exactly the size of axis 1. - `forward_log` - the public entry point that uses the helper - matches the batch mean over its own softmaxed inputs, and is not scaled down by the class axis. - The result stays a scalar, since the reduction change must not alter the shape. With the double reduction restored, the tests fail with the shipped value and the expected one side by side (0.01125 vs 0.09, 0.07822 vs 0.62576 - a factor of 8, the class axis).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
KLCTCLogits._kldivreduces the KL term twice:Every other copy of the same helper reduces once.
ppocr/losses/basic_loss.py:114is the twin, and it carries the contract in a comment:paddle.sum(paddle.mean(v, axis=1)) / n == paddle.sum(v) / (n * m), so the distillation version is smaller than the batch mean by the size of axis 1: a factor of 8 for an 8-way CTC head, and about 6600 for the 6624-character class axisforward_meanloguses.The batch-mean form is the one the repository itself settled on: the helper was introduced by #5643, which validated it against
F.kl_div(..., reduction="batchmean"), and the shipped defaultmode='mean'already goes through the correctkl_loss. So as shipped, the non-default modes disagree with the default by that factor. Axis 1 is the class axis forforward_logand the time axis forforward_meanlog, which is why the extra reduction reads as accidental rather than deliberate.Change
The result stays 0-d and keeps the same
shape[0]denominator, so both call sites (distillation_loss.py:1091and:1149) andDistillCTCLogits.forwardare unaffected.Validation
Executed with PaddlePaddle and the real values: the two reductions differ by exactly the axis-1 size.
Driving the real
KLCTCLogitsclass,mode='log'gives0.081697159as shipped and0.653577with the twin reduction, againstF.kl_div(..., reduction='batchmean')=0.653572(relative error 8e-6, i.e. the twin reduction reproduces the reference and the shipped one does not).mode='meanlog'goes from5.857e-06to0.038804, which is what the shipped defaultmode='mean'already produces (0.038554).black --checkandflake8exit 0 on the file before and after.Honest caveat
No shipped YAML reaches this line today:
DistillCTCLogits.__init__does not forwardmode, and the three configs using the family run the defaultmode='mean', which takes the correct branch. Reaching it needsKLCTCLogits(mode='log')ormode='meanlog'from Python. So this is a latent wrong value rather than a wrong training run today - but it is a wrong value for anyone who selects those modes from code, and the fix is the one-line alignment with both the twin and the shipped default.Competition
All 78 open PRs were enumerated and 64 changed-file lists pulled: none touches
ppocr/losses/. Searches forkldiv,KLCTCLogitsandmeanlogreturn no open item.