r.geomorphon: parallelize with OpenMP using memory-bounded banding - #7783
r.geomorphon: parallelize with OpenMP using memory-bounded banding#7783krcoder123 wants to merge 10 commits into
Conversation
github-actions
Bot
added
raster
|
I attached a benchmarking script in this module as well and I will post my benchmark graphs once the full run finishes on my end. I'll make the pytest for this module in a separate PR this week. |
|
Here are the scaling graphs from my machine (8 core Apple M3, 4 performance and 4 efficiency cores). The speedups and the scalings look pretty good. One note, I had to use a Mac specific benchmark wrapper with cooldowns and pmset because my Mac was throttling a lot using the universal benchmark that's in this PR. So the below images of my graphs are with that setup, but if you would like to also see the throttling graphs done with the universal benchmark I can attach those too. 
|
c3a0552 to
2a92e4f
Compare
|
The parallel pytest cases are written and I tested it locally against this branch. But, it needs the conftest.py file which is already in #7785. Committing with a duplicate here doesn't make sense so I'll wait for that PR to merge first. I'll then rebase and add them here, so the test files don't get duplicated across the two PRs. |
|
I am getting really good scaling on my machine. I think this is good to go once we merge the tests and update the tests here. |
…stead of module globals
The output is processed in bands sized from a new memory option, with the input strip loaded through per thread descriptors and each band's rows computed in parallel. Output is byte identical to the serial module at every thread count across all twelve outputs, both test projections, the memory range, and the option matrix. A new nprocs option matches the other parallel modules.
…mentation Also removes the stale one-off region-size warning; the one-off path now reads only the window rows.
Checks that threaded output stays bit-identical to serial across bands, a mask, the minimum region height, and the extended correction.
819e92c to
ba4aba8
Compare
Thanks for testing the benchmark on your machine. I added the merged tests onto this PR along with the parallel tests so this PR should be good to go now. |
3 participants
This PR parallelizes r.geomorphon with OpenMP while keeping memory usage within the amount the user allows.
The actual parallelization method is similar to r.param.scale parallelization in theory. The output is processed in bands, with the band height computed from a new memory option. For each band the input rows are read in parallel with per thread file descriptors into a shared strip, and each band's rows are then computed in parallel. Cells near the map edges see their search window positioned exactly the way the original code positions its row buffer there, which is what allows the output to be exactly the output serial produced.
Output is exactly the same as the current serial module at 1, 2, 4, and 8 threads, across all twelve output maps, on projected and lat-lon data, through the whole memory range, with masks, and on a 100 million cell map. On my Apple M3 machine with 8 cores, I got a speedup of about 1.9x at 2 threads, 3.3 to 3.7x at 4, and 4.2 to 5.1x at 8.