gSSURGO: Fix spatial sampling and improve data aggregation accuracy#3643
gSSURGO: Fix spatial sampling and improve data aggregation accuracy#3643divine7022 wants to merge 29 commits into
Conversation
infotroph
changed the title
…for custom aoi and switched to single fetchSDA call
This reverts commit 7ca57f6.
dlebauer
left a comment
dlebauer
left a comment
There was a problem hiding this comment.
This looks good to me. Thanks for all of your work on this!
@infotroph it looks like @divine7022 has addressed your suggestions and answered your questions. This seems to be a substantial improvement. Are there any errors that would prevent this from being merged?
infotroph
left a comment
infotroph
left a comment
There was a problem hiding this comment.
A few more comments here. I haven't finished my re-review yet, but I can already say that given the magnitude of the changes it would be very helpful to include some test cases so we can verify the function is behaving as intended.
…ze sand/silt/clay fractions after weighted mean aggregation
…ukey before filtering, add soil_bulk_density to base ensemble, handle single-component soils and gamma sample bulk density
| # Validate depths parameter | ||
| if (depths[1] != 0) { | ||
| PEcAn.logger::logger.severe( | ||
| "First depth must be 0. Use depths = c(0, 0.15, 0.30, ...) like hist() breaks." | ||
| ) | ||
| } |
There was a problem hiding this comment.
Per conversation (much) earlier in this PR, this is clarifying and simplifying behavior that already existed. 👍 to that.
But! This function is the only PEcAn soil tool I can find that expects top-of-layer depths; the others either don't say or expect bottom-of-layer depth:
standard_vars.csvis explicit that depth means to bottom- The SoilGrids tools use bottom depth when one number is needed, but also are good about keeping labels with explicit top and bottom depths in most places.
soil.unitsdoes not say.soil2netcdfdoesn't state either way (nor do any internals care), but its example uses bottom depthwrite.config.SIPNETexplicitly treats depths as bottom of layer- My currently open PR Rothc: Soil file support + read options from settings file #3788 will make RothC assume depths as bottom of layer if nobody stops me first 😈. Working on that is what made me notice the difference from how this here gSSURGO function works.
See also #3775
| test_that("extract_soil_gssurgo requires depths to start with 0", { | ||
| skip_on_cran() | ||
| skip_on_ci() | ||
| tmp_outdir <- withr::local_tempdir("gssurgo_test_") | ||
|
|
||
| # Disable debugging during error testing | ||
| withr::local_options(error = NULL) | ||
|
|
||
| # Should error when depths doesn't start with 0 | ||
| expect_error( | ||
| extract_soil_gssurgo( | ||
| outdir = tmp_outdir, | ||
| lat = 40.1164, | ||
| lon = -88.2434, | ||
| size = 1, | ||
| radius = 500, | ||
| depths = c(0.15, 0.30) # Missing 0 at start | ||
| ), | ||
| regexp = "First depth must be 0" | ||
| ) | ||
|
|
||
| # Should work when depths starts with 0 | ||
| res <- extract_soil_gssurgo( | ||
| outdir = tmp_outdir, | ||
| lat = 40.1164, | ||
| lon = -88.2434, | ||
| size = 1, | ||
| radius = 500, | ||
| depths = c(0, 0.15, 0.30) # Correct format | ||
| ) | ||
|
|
||
| expect_false(is.null(res)) | ||
| }) |
There was a problem hiding this comment.
Depth validation happens before any API calls, so it's safe to remove the skips from this. Avoiding boilerplate also makes it much easier to see what's actually happening in this test:
| test_that("extract_soil_gssurgo requires depths to start with 0", { | |
| skip_on_cran() | |
| skip_on_ci() | |
| tmp_outdir <- withr::local_tempdir("gssurgo_test_") | |
| # Disable debugging during error testing | |
| withr::local_options(error = NULL) | |
| # Should error when depths doesn't start with 0 | |
| expect_error( | |
| extract_soil_gssurgo( | |
| outdir = tmp_outdir, | |
| lat = 40.1164, | |
| lon = -88.2434, | |
| size = 1, | |
| radius = 500, | |
| depths = c(0.15, 0.30) # Missing 0 at start | |
| ), | |
| regexp = "First depth must be 0" | |
| ) | |
| # Should work when depths starts with 0 | |
| res <- extract_soil_gssurgo( | |
| outdir = tmp_outdir, | |
| lat = 40.1164, | |
| lon = -88.2434, | |
| size = 1, | |
| radius = 500, | |
| depths = c(0, 0.15, 0.30) # Correct format | |
| ) | |
| expect_false(is.null(res)) | |
| }) | |
| test_that("extract_soil_gssurgo requires depths to start with 0", { | |
| expect_error( | |
| extract_soil_gssurgo( | |
| outdir = withr::local_tempdir(), | |
| lat = 40, | |
| lon = -88, | |
| depths = c(0.15, 0.30) # Missing 0 at start | |
| ), | |
| regexp = "First depth must be 0" | |
| ) | |
| }) |
(Personally I'd also move this to the tests for gssurgo_fetch_area and keep this file focused on testing the ensemble fitting portions, possibly using mocked results from gssurgo_fetch_area to do it)
| expect_type(res_small, "list") | ||
| expect_type(res_large, "list") |
There was a problem hiding this comment.
Note that this doesn't actually check that the change in buffer had any effect! Am I right we should expect re_larges$mukey_counts to sum to more pixels than res_small$mukey_counts?
There was a problem hiding this comment.
No, I'm thinking of gssurgo_fetch_area.... But this test would give about the same information if it called gssurgo_test_area directly.
| if (requireNamespace("ncdf4", quietly = TRUE) && length(res) >= 3) { | ||
| # Compare two different ensemble members (skip first - it's unsampled) | ||
| nc1 <- ncdf4::nc_open(unlist(res)[2]) | ||
| nc2 <- ncdf4::nc_open(unlist(res)[3]) |
There was a problem hiding this comment.
It's possible you could reduce boilerplate by using soil_ic_netcdf2list() as a helper to read the nc files
| expect_false(is.null(res)) | ||
| expect_type(res, "list") |
There was a problem hiding this comment.
I'm fairly sure testing if the result is a list is also an effective check that it's not null. Not a big deal in one place, but across many test blocks this adds up to a lot of extra lines to scroll through without much added information about the behavior of the function.
| expect_false(is.null(res)) | |
| expect_type(res, "list") | |
| expect_type(res, "list") |
| "mukey", "cokey") %in% names(result$soilprop))) | ||
|
|
||
| # Values should be in original units (percentages, not fractions) | ||
| expect_true(all(result$soilprop$sandtotal_r <= 100, na.rm = TRUE)) |
There was a problem hiding this comment.
Note that this test won't fail if they are fractions! Would checking that sandtotal+silttotal+claytotal == 100 be too strict?
| - `extract_soil_gssurgo()` (#3643) | ||
| - Replaced point-based WFS queries with raster-based WCS approach using `soilDB::mukey.wcs()`. | ||
| - Replaced `grid_size`/`grid_spacing` parameters with `radius` (meters) for simpler buffer-based AOI creation. | ||
| - Switched to single `soilDB::fetchSDA()` call for component-level soil data retrieval, enabling better ensemble uncertainty quantification through within-map-unit variability. Added support for custom AOI polygons. | ||
| - This eliminates spatial coverage gaps and reduces network requests while maintaining backward compatibility. |
|
What is the status of this PR? Also, Does this pr #3455 overlap or augment other work that has been/will be done on the gssurgo handling? |
4 participants
Description
Refactored
extract_soil_gssurgo()to replace point-based WFS queries with raster-based Web Coverage Service (WCS) approach, enabling accurate area-weighted sampling and eliminating spatial coverage gaps.soilDB::mukey.wcs()to extract map unit keys with complete spatial coverage at 30m resolution. This provides accurate area weighting through pixel counts and eliminates spatial coverage gaps.soilDB::get_SDA_property()with "Weighted Average" aggregation method. This retrieves soil properties (sand, silt, clay, organic matter, bulk density) integrated across specified depth ranges with component weighting in a single batch query.soilDB::fetchSDA()to obtain complete rock fragment data (fragvol_r) representing total volume across all size classes: 2-75mm (pebbles), 75-250mm (cobbles), 250-600mm (stones), and >600mm (boulders). Applied proper depth and component weighting for fragments:Motivation and Context
Fixes #3609
Review Time Estimate
Types of changes
Checklist: