Skip to content

[Perf] Vectorise WarpMLS used by the TIA augmentations (6–9× faster) - #18422

Open
ErshovVE wants to merge 3 commits into
PaddlePaddle:mainfrom
ErshovVE:perf/warp-mls-vectorise
Open

ErshovVE wants to merge 3 commits into
PaddlePaddle:mainfrom
ErshovVE:perf/warp-mls-vectorise

Conversation

@ErshovVE

Copy link
Copy Markdown

Motivation

RecAug applies the TIA augmentations (tia_distort, tia_stretch, tia_perspective). All three warp the image
with WarpMLS (moving least squares), which is written with Python loops:

  • calc_delta loops over grid nodes and control points with scalar numpy operations;
  • gen_img loops over grid cells.

When we profiled the data loading of PP-OCRv6 small rec training (RecAug + RecConAug), TIA took ~73 % of the
time, and one CPU core gave 11 samples/s. On wide text lines the data loader, not the GPU, set the training speed.

Change

ppocr/data/imaug/text_image_aug/warp_mls.py, same class and the same public interface (WarpMLS(src, src_pts, dst_pts, dst_w, dst_h, trans_ratio).generate()):

  • the displacements of all grid nodes are solved at once with numpy broadcasting;
  • they are interpolated to every pixel separably (bilinear, the same weights as the loop);
  • the image is sampled once with cv2.remap (INTER_LINEAR, BORDER_REPLICATE).

The special cases of the loop are kept: a grid node that coincides with a control point snaps to it; with fewer
than two points the image is returned unwarped.

Speed

One WarpMLS.generate() on TIA distort points (segment 6, the maximum RecAug uses), 3-channel 48 px high
images, CPU, the same points for both versions:

Image (h × w) Loop, ms Vectorised, ms Speed-up
48 × 150 4.0 0.68 5.8×
48 × 320 7.5 0.84 8.9×
48 × 600 11.9 1.75 6.8×
48 × 1200 22.9 2.68 8.6×
48 × 2400 45.1 6.01 7.5×

Output

  • The displacement field equals the loop's algorithm run in float64 to 1e-6 px, at widths up to 4 000 px.
  • The current loop accumulates in float32 and drifts with the coordinate size: it stays within a small tolerance of
    the new field up to 1 600 px, and differs by up to ~0.5 px near the right edge of ~3 000 px wide images.
  • Images: cv2.remap quantises sub-pixel positions to 1/32 px, so about half of the pixel values differ from the loop
    version by one level and none by more than two. In the timing above the maximum difference was 1 level.

Tests

tests/ppocr/test_warp_mls.py keeps the original loop implementation as the reference (in float32, and in float64
as the exact one):

  • the displacement field against the float64 loop at sizes from 16 × 30 to 44 × 4000, including corner points
    that coincide with grid nodes;
  • tia_distort, tia_stretch and tia_perspective with the old and the new WarpMLS, the same random seed, 1
    and 3 channels: images within 2 levels, mean difference below 0.75;
  • fewer than two control points: the image is left unwarped.
pytest tests/ppocr/test_warp_mls.py      # 23 passed

calc_delta looped over grid nodes and control points with scalar numpy
ops, and gen_img looped over grid cells: in RecAug + RecConAug the TIA
warp was ~73 % of the data-loading time (11 samples/s per CPU core on
PP-OCRv6 small RU/EN data). Now the node displacements are solved for
all nodes at once, interpolated to pixels separably and sampled with
cv2.remap: 6-11x faster per call (48x150..48x1200), the whole training
data pipeline 3x faster.

The output matches the loop version (kept in the test as reference):
node displacements within 0.05 px (the loop accumulated in float32),
images within a level or two on smooth images and up to ~5 levels on
sharp text edges (remap quantises sub-pixel positions to 1/32 px and
rounds instead of truncating).
…4000 px

On wide crops (~3000 px, synthetic lines up to 250 chars) the original
float32 loop drifts by up to ~0.5 px near the right edge, so a few pixels
of sharp glyph edges differed by up to 33 levels from the vectorised
version. The vectorised field equals the same loop run in float64 to
1e-10 px; the test now compares against that reference strictly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant