docs: Hydra admission design (accuracy, cost, candidates, sketch-bench needs)

[zzylol]

Builds on #190's cost table (docs/rqe_optimizer_cost_model.md §3). Docs and comments only; no behavior change.

Adds docs/rqe_optimizer_hydra.md, the design for admitting Hydra into the rqe-optimizer, and links it from the other optimizer docs. Hydra is not a candidate today: hydra-kll has FamilyProperties but is in no capability's family list, and the other four variants have none.

Contents

1. Hydra as sketch-bench runs it (asap_sketchlib 0.3.0):
   • an R × W grid;
   • every record is inserted into all 2^d − 1 label subsets;
   • the answer is the median over rows;
   • no keys are stored, and merges are cell by cell;
   • the wrapper only queries prefix groupings;
   • per-variant footprints for hydra-cms, cs, hll, kll and univmon.
2. Accuracy per combination. These are derived bounds, checked against the paper's Theorem 2 (VLDB 2022, arXiv:2208.04927) and against the library.
   • Contamination is additive in the global fanned-out mass, β·M/W.
   • Inner errors are correlated across rows, because every cell shares seeds. The confidence is therefore P[Bin(R, 1−1/β) > R/2] − R·δ_in, about 0.69 at the canonical R = 3.
   • The column hash has one fixed seed. hydra-univmon's 2 × 8 grid makes even the form of its bound vacuous.
   • CMS, CountSketch and HLL have closed forms that are vacuous for small groups. UnivMon has a form only. KLL and entropy have none. So admission has to be empirical, via a measured predicate.
3. Cost model. The cost doc's §3 table with I_d = I_r = 1, c_probe,h(G_r) per probe, and the key tracker as a part.
4. Candidates and MILP.
   • The MILP is unchanged.
   • Candidates are keyed by an ordered schema Λ.
   • An edge exists for ∅ ≠ G_i ⊆ Λ when a measurement covers it.
   • Accuracy is a property of each edge, so the MILP stays linear.
5. What sketch-bench must add (10 items). hydra-kll is measured in the same rerun as the other four variants.
6. Open decisions (6).

Consolidated with what was already on main (second commit)

• docs/rqe_optimizer_cost_model.md:
  • The Shared + Fixed bullet no longer calls card(G_d) ≤ subpopulations Hydra's rule, or fan-out a deviation. That check is necessary, not sufficient: a group's error depends on the grid's fanned-out mass against the group's own size, so admission is §2.4's measured predicate.
  • The HydraKLL column of the side-by-side table moves to Hydra §3.
  • The key tracker's grouping is marked open (§6.1).
• candidates.rs: measured_at_group_count's comment says the same. Behavior is unchanged, because HydraKLL is never a candidate.
• configs/datagen/hydra_columns.yaml and hydra_kll_disjoint_labels.yaml: since asap_sketchlib 0.3.0, the sketch keys subsets by column, and only the polars baseline still aliases (Overlapping label-column domains alias into one Hydra cell, and the comparator scores it as sketch error #74).
• Issues closed in favor of this design: rqe-optimizer: support shared multi-subpopulation sketches (HydraKLL) in the cost model #142 (superseded), bench: measure hydra-kll without subset fan-out (one key per grouping) #165 (its no-fan-out scope contradicts §5), rqe-optimizer: hydra-kll accuracy when a query merges several windows #166 (duplicate of §5.6).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis