Skip to content

Open Building Map as a feature, and as the residential split for Microsoft v8 - #21

Open
MFStark wants to merge 4 commits into
mainfrom
feature/mfiking/obm_feature
Open

Open Building Map as a feature, and as the residential split for Microsoft v8#21
MFStark wants to merge 4 commits into
mainfrom
feature/mfiking/obm_feature

Conversation

@MFStark

@MFStark MFStark commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Adds Open Building Map (OBM) as a model feature, and a microsoft_v8_obm
product that uses it as Microsoft v8's residential split in place of GHSL.

Why

Footprint datasets record where buildings are, not what they are for — a
warehouse and an apartment block look identical from above. Every current
product multiplies total building volume by a proportion_residential that
comes from GHSL, a satellite classification. Nothing in the stack observes
building use.

OBM does: 2.69 billion buildings, each carrying an occupancy type from the GEM
Building Taxonomy.

What this adds

OBM features (obm_20250404_*) — six measures per block, collapsed from the
covariate's eight occupancy classes:

measure
density sum of the eight parent classes, rescaled where they overlap
height GHSL ANBH, one storey (2.5 m) imputed where OBM sees a footprint and GHSL sees none
volume, residential_volume derived
proportion_residential 1 − nonres/total, with unknown counted as residential
p_observed binary: did OBM observe this fraction, or was it imputed?

microsoft_v8_obm_20250404_residential_volume — Microsoft's own density and
height, untouched; only the residential fraction is swapped:

p = obm_proportion_residential   where OBM sees a building
    ghsl_proportion_residential  where only GHSL does
    1.0                          where neither does

Results

Feeding GHSL alone through the same chain reproduces the shipped
microsoft_v8_residential_volume to 5e-07. That exactness is what makes the
swap attributable — a difference cannot be our arithmetic.

Globally, across 4,797 blocks and 2.11B built pixels:

  • +60.4 km³ of non-residential building volume — 1.54× GHSL's total
    (111.6 → 172.1 km³, moving 10.6% → 16.4% of all v8 building)
  • OBM supplies the fraction on 48.6% of v8's built pixels but 62.5% of its
    volume
    — its pixels carry 1.29× the average building density
  • 17.8% of pixels reach neither source and take p = 1, but those carry only
    5.7% of the volume

An earlier two-city pilot predicted 1.6×–3.4×. The global answer landing just
below that range is what you would expect: those cities are where OBM's
labelling is unusually complete.

Two things reviewers should know

Every non-residential figure is a floor. 61% of OBM's buildings carry no
occupancy label and our convention counts them as residential. Each one tagged
later can only move the number one way.

Height is not independent of GHSL. Both sides use GHSL's ANBH, so a volume
comparison reduces to area × use. The residential split is independent — it
comes from OSM tags.

Notes

  • p_observed exists because proportion_residential cannot carry this: OBM's
    own p = 1 and the no-source fallback write the same number. On one test
    block 98% of OBM's pixels sit at p = 1 already.
  • Only residential_volume is written for microsoft_v8_obm. The spliced
    fraction is recoverable as residential_volume / microsoft_v8_volume.
  • Provider renamed obm_2025q2obm_20250404, after the GFZ upload the data
    came from rather than the quarter we filed it under.

Marcon Louie Fikingas and others added 4 commits August 12, 2026 10:22
Microsoft v8 supplies its own density and height but borrows a residential
fraction from GHSL. This adds a provider that takes that fraction from OBM
where OBM sees a building, falls back to GHSL where it does not, and to
fully-residential where neither does, so the layer stays global while every
pixel that differs from microsoft_v8 differs because OBM said something -
not because it said nothing.

Feeding GHSL alone through the same chain reproduces the shipped v8 raster to
5e-07, which is what makes the swap attributable rather than an artefact of
our arithmetic. Globally the product identifies 60 km3 more non-residential
building volume than GHSL - 1.54x its total, moving 10.6% to 16.4% of all v8
building.

Only residential_volume is written. The spliced fraction is recoverable as
residential_volume / microsoft_v8_volume, and the observation mask describes
OBM rather than the splice, so it is written by the OBM build instead - one
raster for a static snapshot rather than 24 near-identical copies.

Also adds obm_*_p_observed, which marks where OBM actually observed the
fraction. Without it a consumer cannot tell a measured value from an imputed
one: OBM's own p = 1 and the no-source fallback write the same number, and on
one test block 98% of OBM's pixels sit at p = 1 already.

The provider is renamed obm_2025q2 -> obm_20250404, after the GFZ upload the
data came from rather than the quarter we filed it under.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OBM was a bespoke feature builder that hand-rolled its own writing and its own
symlink fan-out alongside the built-version machinery every other provider
uses. It is now a registered BuiltVersion with an ObmStrategy that overrides
only the two things genuinely specific to it: reading eight occupancy classes
from a different root, and opting out of the spatially-averaged companions it
does not produce.

Registering it as a single-epoch version means get_processing_strategy sees a
time point that is both first and last, so it fills the other 69 by symlink
automatically. The static-snapshot layout now falls out of the existing
machinery rather than being reimplemented - 414 links per block, verified.

BuiltVersion gains two optional fields, both defaulting so no existing version
changes behaviour: `strategy` names the class that builds a version's own
measures, and `geospatial_averages` gates the averaged companions.

_generate_microsoft_derived_measures is renamed to
_generate_provider_derived_measures and its spec dict lifted to module scope,
since it is no longer Microsoft-only in principle. GHSL and the four Microsoft
versions produce byte-identical rasters - verified against the shipped v8.
The provider dispatch gains an explicit NO_DERIVED_MEASURES set, which keeps
the original ValueError on an unrecognised version rather than silently
producing nothing.

ObmStrategy writes all six measures rather than delegating volume and
residential_volume to the shared derivation. derive_features computes them
anyway so check_features can assert volume / density == height, and deriving
them again from the written float32 rasters rounds at a different point - a
1 ulp difference on ~0.05% of land pixels. Writing what was already validated
keeps this a pure refactor.

obm.py loses process_obm and link_features, ~117 lines: it is now compute
only, and ObmStrategy owns writing. obm_features stays as a thin wrapper over
the same strategy, so the targeted command and the general one cannot produce
different rasters.

Verified byte-identical on two blocks spanning 10% and 92% height imputation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Make OBM a registered built version instead of a special case
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant