Skip to content

polars: tag Hydra baseline subset keys by column (#74) - #195

Merged
zzylol merged 2 commits into
mainfrom
74-polars-tagged-subkeys
Oct 11, 2026
Merged

zzylol merged 2 commits into
mainfrom
74-polars-tagged-subkeys

Conversation

@zzylol

@zzylol zzylol commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #74. This is PR 1 of the Hydra measurement plan in #193 (docs/rqe_optimizer_hydra.md §5.2).

Problem. asap_sketchlib 0.3.0's Hydra keys each label subset by column (label0:abc;label1:x), so equal values in different label columns never alias in the sketch. The polars exact baselines still built 0.2.2's untagged key (abc;x). On label columns that share a value domain, such as configs/datagen/hydra_columns.yaml's shared alphabet, they merged a key1 group with the key2 group of the same value. As a result the exact rows scored nonzero.

Change

  • polars_shared::subset_key builds the 0.3.0 subkey over hydra_shared's label{i} schema, escaping \, : and ; as the library does.
  • prefix_key is the lookup for a probe of the leading columns, and every polars lookup uses it: frequency, cardinality, univmon and hydra-kll.
  • An empty label keeps its column, as 0.3.0 does. The old key dropped it.
  • The two datagen yaml headers no longer describe the aliasing as current.

Tests

  • subset_keys_tag_each_value_with_its_column.
  • a_value_shared_by_two_columns_is_counted_once: records a;a and a;b. Group a in column 0 holds 2 records; the old key counted 3.
  • cargo test -p sketch-bench passes (98), as do fmt and clippy.

Evidence. These are the polars rows on the shared-alphabet data: hydra_columns.yaml, and its _f64 version for KLL. Grids: cms 3×128 with 3×512 cells, hll 3×128, kll 3×64 with k=200. --operations query --metrics accuracy.

Row main (f9c4b68) this branch
hydra-cms polars are_all 0.596, aae_all 3.74 0, 0
hydra-hll polars are_all 0.249, aae_all 56.8 0, 0
hydra-kll polars mean_rank_err 0.00093, max_rank_err 0.036 0, 0

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

zzylol and others added 2 commits October 11, 2026 16:35
The polars exact baselines keyed a label subset by its `;`-joined values,
asap_sketchlib 0.2.2's subkey. 0.3.0's Hydra tags each value with its
column (`label0:a;label1:b`), so on label columns that share a value
domain the baseline merged a key1 group with the same key2 group and
scored nonzero against itself. subset_key now builds the 0.3.0 key over
hydra_shared's `label{i}` schema (escaping `\`, `:` and `;`), empty
labels keep their column, and every lookup goes through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol
zzylol force-pushed the 74-polars-tagged-subkeys branch from 912c3ab to 4149b7e Compare October 11, 2026 16:35
@zzylol
zzylol merged commit 2e1cd9c into main Oct 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overlapping label-column domains alias into one Hydra cell, and the comparator scores it as sketch error

1 participant