perf: optionally dictionary-encode geog_id in the Arrow mapping output

[mailletf]

Problem

expand_geohash_mapping_arrow emits one row per (geog_id, geohash) pair, so every row of a group repeats that group's id. The geog_id column is almost entirely duplicate bytes.

Fix

Add dictionary_geog_id, which emits Dictionary<Int32, Utf8> instead: one copy of each id in the dictionary, 4 bytes of index per row.

Defaults to false, so the existing schema is unchanged for current callers.

Benchmarks

20,000 geographies × 300 cells = 6M rows, with short 11-character ids:

	plain	dictionary
geog_id column	114.0 MB	24.3 MB	4.7×
whole batch	204.0 MB	114.3 MB	1.8×
call	362 ms	294 ms	1.2×

Longer ids widen the gap — the dictionary column's size does not depend on id length beyond the one stored copy.

Considered

Run-end encoding would compress far harder still (the rows are already grouped, so 6M rows is really 20,000 runs), but support outside pyarrow is patchier than for dictionaries, so it is not offered here.

Also

Stops threading the geog_id String through the parallel expansion, which never needed it: results come back in input order, so results[i] belongs to geog_ids[i].

[mailletf]

This change is part of the following stack:

• fix: don't drop multipolygon parts whose seed lands in an earlier part #30
  • perf: cover polygons by descending the geohash tree #31
    • perf: optionally dictionary-encode geog_id in the Arrow mapping output #39 ◀
      • perf: cover large polygons across Rayon workers #40
        • fix: reject null list slots and empty geohashes #42

_{Change managed by git-spice.}

[Copilot]

Pull request overview

This PR adds an optional dictionary_geog_id flag to expand_geohash_mapping_arrow to reduce Arrow output size by dictionary-encoding the geog_id column while preserving the existing default schema for current callers.

Changes:

• Added dictionary_geog_id (default false) to optionally emit geog_id as Dictionary<Int32, Utf8> in the Arrow RecordBatch output.
• Refactored the parallel expansion path to avoid threading geog_id through Rayon work items (re-associating via preserved result order).
• Added Python tests and updated the Python type stub to cover/describe the new optional argument and schema behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/lib.rs	Implements optional dictionary encoding for geog_id and refactors parallel expansion result handling.
tests/test_arrow_ops.py	Adds test coverage for dictionary-encoded geog_id equivalence, schema typing, size reduction, and edge cases.
geohash_polygon/__init__.pyi	Updates the Python stub signature and docstring to include dictionary_geog_id.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mailletf]

Agreed, fixed in bbd2a02. expand_geohash_mapping_arrow now checks the group count against i32::MAX before building the keys and raises a ValueError naming the limit and the way out (dictionary_geog_id=False, or split the input). A wrapped negative key would silently repoint rows at the wrong geog_id, which is the worst kind of failure here.