Open Building Map as a feature, and as the residential split for Microsoft v8 - #21
Open
MFStark wants to merge 4 commits into
Open
Open Building Map as a feature, and as the residential split for Microsoft v8#21MFStark wants to merge 4 commits into
MFStark wants to merge 4 commits into
Conversation
Microsoft v8 supplies its own density and height but borrows a residential fraction from GHSL. This adds a provider that takes that fraction from OBM where OBM sees a building, falls back to GHSL where it does not, and to fully-residential where neither does, so the layer stays global while every pixel that differs from microsoft_v8 differs because OBM said something - not because it said nothing. Feeding GHSL alone through the same chain reproduces the shipped v8 raster to 5e-07, which is what makes the swap attributable rather than an artefact of our arithmetic. Globally the product identifies 60 km3 more non-residential building volume than GHSL - 1.54x its total, moving 10.6% to 16.4% of all v8 building. Only residential_volume is written. The spliced fraction is recoverable as residential_volume / microsoft_v8_volume, and the observation mask describes OBM rather than the splice, so it is written by the OBM build instead - one raster for a static snapshot rather than 24 near-identical copies. Also adds obm_*_p_observed, which marks where OBM actually observed the fraction. Without it a consumer cannot tell a measured value from an imputed one: OBM's own p = 1 and the no-source fallback write the same number, and on one test block 98% of OBM's pixels sit at p = 1 already. The provider is renamed obm_2025q2 -> obm_20250404, after the GFZ upload the data came from rather than the quarter we filed it under. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OBM was a bespoke feature builder that hand-rolled its own writing and its own symlink fan-out alongside the built-version machinery every other provider uses. It is now a registered BuiltVersion with an ObmStrategy that overrides only the two things genuinely specific to it: reading eight occupancy classes from a different root, and opting out of the spatially-averaged companions it does not produce. Registering it as a single-epoch version means get_processing_strategy sees a time point that is both first and last, so it fills the other 69 by symlink automatically. The static-snapshot layout now falls out of the existing machinery rather than being reimplemented - 414 links per block, verified. BuiltVersion gains two optional fields, both defaulting so no existing version changes behaviour: `strategy` names the class that builds a version's own measures, and `geospatial_averages` gates the averaged companions. _generate_microsoft_derived_measures is renamed to _generate_provider_derived_measures and its spec dict lifted to module scope, since it is no longer Microsoft-only in principle. GHSL and the four Microsoft versions produce byte-identical rasters - verified against the shipped v8. The provider dispatch gains an explicit NO_DERIVED_MEASURES set, which keeps the original ValueError on an unrecognised version rather than silently producing nothing. ObmStrategy writes all six measures rather than delegating volume and residential_volume to the shared derivation. derive_features computes them anyway so check_features can assert volume / density == height, and deriving them again from the written float32 rasters rounds at a different point - a 1 ulp difference on ~0.05% of land pixels. Writing what was already validated keeps this a pure refactor. obm.py loses process_obm and link_features, ~117 lines: it is now compute only, and ObmStrategy owns writing. obm_features stays as a thin wrapper over the same strategy, so the targeted command and the general one cannot produce different rasters. Verified byte-identical on two blocks spanning 10% and 92% height imputation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Make OBM a registered built version instead of a special case
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Open Building Map (OBM) as a model feature, and a
microsoft_v8_obmproduct that uses it as Microsoft v8's residential split in place of GHSL.
Why
Footprint datasets record where buildings are, not what they are for — a
warehouse and an apartment block look identical from above. Every current
product multiplies total building volume by a
proportion_residentialthatcomes from GHSL, a satellite classification. Nothing in the stack observes
building use.
OBM does: 2.69 billion buildings, each carrying an occupancy type from the GEM
Building Taxonomy.
What this adds
OBM features (
obm_20250404_*) — six measures per block, collapsed from thecovariate's eight occupancy classes:
densityheightvolume,residential_volumeproportion_residential1 − nonres/total, withunknowncounted as residentialp_observedmicrosoft_v8_obm_20250404_residential_volume— Microsoft's own density andheight, untouched; only the residential fraction is swapped:
Results
Feeding GHSL alone through the same chain reproduces the shipped
microsoft_v8_residential_volumeto 5e-07. That exactness is what makes theswap attributable — a difference cannot be our arithmetic.
Globally, across 4,797 blocks and 2.11B built pixels:
(111.6 → 172.1 km³, moving 10.6% → 16.4% of all v8 building)
volume — its pixels carry 1.29× the average building density
p = 1, but those carry only5.7% of the volume
An earlier two-city pilot predicted 1.6×–3.4×. The global answer landing just
below that range is what you would expect: those cities are where OBM's
labelling is unusually complete.
Two things reviewers should know
Every non-residential figure is a floor. 61% of OBM's buildings carry no
occupancy label and our convention counts them as residential. Each one tagged
later can only move the number one way.
Height is not independent of GHSL. Both sides use GHSL's ANBH, so a volume
comparison reduces to area × use. The residential split is independent — it
comes from OSM tags.
Notes
p_observedexists becauseproportion_residentialcannot carry this: OBM'sown
p = 1and the no-source fallback write the same number. On one testblock 98% of OBM's pixels sit at
p = 1already.residential_volumeis written formicrosoft_v8_obm. The splicedfraction is recoverable as
residential_volume / microsoft_v8_volume.obm_2025q2→obm_20250404, after the GFZ upload the datacame from rather than the quarter we filed it under.