Repository navigation
Conversation
calc_delta looped over grid nodes and control points with scalar numpy ops, and gen_img looped over grid cells: in RecAug + RecConAug the TIA warp was ~73 % of the data-loading time (11 samples/s per CPU core on PP-OCRv6 small RU/EN data). Now the node displacements are solved for all nodes at once, interpolated to pixels separably and sampled with cv2.remap: 6-11x faster per call (48x150..48x1200), the whole training data pipeline 3x faster. The output matches the loop version (kept in the test as reference): node displacements within 0.05 px (the loop accumulated in float32), images within a level or two on smooth images and up to ~5 levels on sharp text edges (remap quantises sub-pixel positions to 1/32 px and rounds instead of truncating).
…4000 px On wide crops (~3000 px, synthetic lines up to 250 chars) the original float32 loop drifts by up to ~0.5 px near the right edge, so a few pixels of sharp glyph edges differed by up to 33 levels from the vectorised version. The vectorised field equals the same loop run in float64 to 1e-10 px; the test now compares against that reference strictly.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
RecAugapplies the TIA augmentations (tia_distort,tia_stretch,tia_perspective). All three warp the imagewith
WarpMLS(moving least squares), which is written with Python loops:calc_deltaloops over grid nodes and control points with scalar numpy operations;gen_imgloops over grid cells.When we profiled the data loading of PP-OCRv6 small rec training (
RecAug+RecConAug), TIA took ~73 % of thetime, and one CPU core gave 11 samples/s. On wide text lines the data loader, not the GPU, set the training speed.
Change
ppocr/data/imaug/text_image_aug/warp_mls.py, same class and the same public interface (WarpMLS(src, src_pts, dst_pts, dst_w, dst_h, trans_ratio).generate()):cv2.remap(INTER_LINEAR,BORDER_REPLICATE).The special cases of the loop are kept: a grid node that coincides with a control point snaps to it; with fewer
than two points the image is returned unwarped.
Speed
One
WarpMLS.generate()on TIA distort points (segment6, the maximumRecAuguses), 3-channel 48 px highimages, CPU, the same points for both versions:
Output
the new field up to 1 600 px, and differs by up to ~0.5 px near the right edge of ~3 000 px wide images.
cv2.remapquantises sub-pixel positions to 1/32 px, so about half of the pixel values differ from the loopversion by one level and none by more than two. In the timing above the maximum difference was 1 level.
Tests
tests/ppocr/test_warp_mls.pykeeps the original loop implementation as the reference (in float32, and in float64as the exact one):
that coincide with grid nodes;
tia_distort,tia_stretchandtia_perspectivewith the old and the newWarpMLS, the same random seed, 1and 3 channels: images within 2 levels, mean difference below 0.75;