Skip to content

[BugFix] Make KLCTCLogits._kldiv a batch-mean loss like its twin - #18390

Open
MincongZhou wants to merge 2 commits into
PaddlePaddle:mainfrom
MincongZhou:fix/kldiv-batch-mean
Open

MincongZhou wants to merge 2 commits into
PaddlePaddle:mainfrom
MincongZhou:fix/kldiv-batch-mean

Conversation

@MincongZhou

Copy link
Copy Markdown

Problem

KLCTCLogits._kldiv reduces the KL term twice:

# ppocr/losses/distillation_loss.py:1108
loss = target * (paddle.log(target + eps) - x)
loss = paddle.sum(paddle.mean(loss, axis=1)) / loss.shape[0]

Every other copy of the same helper reduces once. ppocr/losses/basic_loss.py:114 is the twin, and it carries the contract in a comment:

loss = target * (paddle.log(target + eps) - x)
# batch mean loss
loss = paddle.sum(loss) / loss.shape[0]

paddle.sum(paddle.mean(v, axis=1)) / n == paddle.sum(v) / (n * m), so the distillation version is smaller than the batch mean by the size of axis 1: a factor of 8 for an 8-way CTC head, and about 6600 for the 6624-character class axis forward_meanlog uses.

The batch-mean form is the one the repository itself settled on: the helper was introduced by #5643, which validated it against F.kl_div(..., reduction="batchmean"), and the shipped default mode='mean' already goes through the correct kl_loss. So as shipped, the non-default modes disagree with the default by that factor. Axis 1 is the class axis for forward_log and the time axis for forward_meanlog, which is why the extra reduction reads as accidental rather than deliberate.

Change

-        loss = paddle.sum(paddle.mean(loss, axis=1)) / loss.shape[0]
+        # batch mean loss
+        loss = paddle.sum(loss) / loss.shape[0]

The result stays 0-d and keeps the same shape[0] denominator, so both call sites (distillation_loss.py:1091 and :1149) and DistillCTCLogits.forward are unaffected.

Validation

Executed with PaddlePaddle and the real values: the two reductions differ by exactly the axis-1 size.

axis-1 size                       = 8
shipped reduction (this line)     = -0.0112499986
twin reduction (basic_loss.py)    = -0.0899999961
twin / shipped                    = 8.0000

Driving the real KLCTCLogits class, mode='log' gives 0.081697159 as shipped and 0.653577 with the twin reduction, against F.kl_div(..., reduction='batchmean') = 0.653572 (relative error 8e-6, i.e. the twin reduction reproduces the reference and the shipped one does not). mode='meanlog' goes from 5.857e-06 to 0.038804, which is what the shipped default mode='mean' already produces (0.038554).

black --check and flake8 exit 0 on the file before and after.

Honest caveat

No shipped YAML reaches this line today: DistillCTCLogits.__init__ does not forward mode, and the three configs using the family run the default mode='mean', which takes the correct branch. Reaching it needs KLCTCLogits(mode='log') or mode='meanlog' from Python. So this is a latent wrong value rather than a wrong training run today - but it is a wrong value for anyone who selects those modes from code, and the fix is the one-line alignment with both the twin and the shipped default.

Competition

All 78 open PRs were enumerated and 64 changed-file lists pulled: none touches ppocr/losses/. Searches for kldiv, KLCTCLogits and meanlog return no open item.

ppocr/losses/distillation_loss.py:1108 reduced the KL term twice:

    loss = paddle.sum(paddle.mean(loss, axis=1)) / loss.shape[0]

The other copy of the same helper (ppocr/losses/basic_loss.py:114) reduces
once and says so:

    # batch mean loss
    loss = paddle.sum(loss) / loss.shape[0]

so the distillation version is smaller by the size of axis 1 - a factor of 8
for a 8-way CTC head, and ~6600x for the 6624-character class axis that
`mode='meanlog'` uses. The batch-mean form is the one that matches the
reference: the helper was introduced by PaddlePaddle#5643 and validated there against
`F.kl_div(..., reduction="batchmean")`, and the shipped default
`mode='mean'` already uses the correct `kl_loss`, so as shipped the non-default
modes disagree with the default by that factor.

Axis 1 is the class axis for `forward_log` and the time axis for
`forward_meanlog`, which is why the extra reduction is accidental rather than
deliberate.

Executed with the real class and PaddlePaddle: the two reductions differ by
exactly the axis-1 size (ratio 8.0), and the result stays 0-d, so both call
sites (distillation_loss.py:1091 and :1149) and DistillCTCLogits.forward are
unaffected.
Adds tests/losses/test_distillation_loss.py, which pins the reduction this PR
restores:

- `_kldiv` equals the single reduction `sum(term) / term.shape[0]`, and not the
  double reduction `sum(mean(term, axis=1)) / term.shape[0]`; the two differ by
  exactly the size of axis 1.
- `forward_log` - the public entry point that uses the helper - matches the
  batch mean over its own softmaxed inputs, and is not scaled down by the class
  axis.
- The result stays a scalar, since the reduction change must not alter the
  shape.

With the double reduction restored, the tests fail with the shipped value and
the expected one side by side (0.01125 vs 0.09, 0.07822 vs 0.62576 - a factor
of 8, the class axis).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant