Skip to content

datagen: child_of columns for hierarchical labels - #197

Merged
zzylol merged 3 commits into
mainfrom
datagen/child-of
Oct 11, 2026
Merged

zzylol merged 3 commits into
mainfrom
datagen/child-of

Conversation

@zzylol

@zzylol zzylol commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • aqpbm-datagen: one new column kind. child_of: <index> with fan_out: f nests a string column under an earlier string column: value = parent row's value + . + child index in [0, f). The index is the column's own distribution draw (0-based rank in its domain), so the existing skew/seed options pick it, independently of the parent.
  • Refused before drawing: child_of without fan_out or the reverse; child_of naming itself, a later column or no column; a non-string parent or child; fan_out: 0; fan_out disagreeing with the distribution's domain size (or an unbounded distribution); cardinality or a string: block on a child.
  • configs/datagen/hydra_hier.yaml: region 4 → service fan-out 25 → endpoint fan-out 25, an independent status label (4, Zipf 2.0), then an i64 value column (user_id, 1e6 Zipf 0.8). Same layout the hydra rows read: label columns (joined with ;) then the value column.
  • docs/aqpbm-datagen.md: a child_of section. ColumnSpec gains two optional fields, so the struct literals elsewhere get child_of: None, fan_out: None and the checked-in record schema is regenerated.

Why

Hydra study/eval work (§6.3) needs hierarchical label schemas (schema width d, nested groupings).

Tests

aqpbm-datagen (45 pass): determinism; every child value is <parent>.<i> with i < fan_out; distinct children per parent ≤ fan_out; the child index equals the column's own ranked draw, and Zipf(1.2) head share lands in its expected range; hydra_hier.yaml loads and generates 4 regions / 100 services as parents; one table-driven test for every validation error. Each was checked by mutation (index /2, dropped domain check). cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo test --workspace clean.

Evidence

approxbench sketchbench --variant hydra-cms --library lib --dtype i64 \
  --spec configs/datagen/hydra_hier.yaml \
  --config "rows=3 cols=1024 cell_rows=3 cell_cols=1024" \
  --operations query --metrics accuracy
row insert throughput (records/s) query accuracy
hydra-cms / lib 194,462 are_all 35.47, are_top10 0.117, 145,365 probes over 4 regions
hydra-cms / polars 17.8M are_all 0.0
hydra-hll / lib (rows=3 cols=1024) 212,090 relative_error_mean 0.0177
hydra-hll / polars 15.7M 0.0

(insert numbers from the same command with --operations insert --metrics throughput; release build, 200k records, 15 cell insertions each.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

zzylol and others added 3 commits October 10, 2026 04:11
A `child_of: <index>` column with `fan_out: f` nests under an earlier
string column: its value is the parent row's value, a `.`, and a child
index in [0, f) drawn from the column's own distribution. Bad indices,
a fan_out that disagrees with the domain, and fields that would not mean
what they say on a child are refused before drawing.

Adds configs/datagen/hydra_hier.yaml (region 4 -> service 25 -> endpoint
25, status 4, user_id value) for the hydra rows, docs, and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
…cuts

A uniform draw is continuous, so `uniform{0.0, 2.9}` with fan_out 2 drew
child indices 0, 1 and 2; child_of now requires whole uniform bounds.
hydra_hier_d2.yaml and hydra_hier_d3.yaml cut hydra_hier.yaml to its first
2 and 3 labels for the Hydra study's schema-width sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis
@zzylol
zzylol merged commit af0ed4d into main Oct 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant