rqe-optimizer: admit Hydra (HLL, UnivMon cardinality, KLL) as candidates, with measured accuracy by coverage

[zzylol]

Implements PR 7 of the Hydra plan (docs/rqe_optimizer_hydra.md §3–§4, #193) on top of #196, which the runner needs for coverage and schema streams. Reads hydra_saturation.csv in the format PR 6 will write (the "eval-shaped Hydra measurements" amendment).

What changed

• Families (lib.rs): hydra-hll and hydra-univmon-cardinality (Cardinality, relative_error) and hydra-kll (Quantile, mean_rank_err) get FamilyProperties: one fixed-size grid, DeltaSet key tracker, mergeable_across_groups = false, and a new answers_any_subgrouping. They are in Capability::families but not in DEPLOYABLE_FAMILIES. hydra-kll and hydra-univmon-cardinality merge lossily; hydra-hll merges exactly.
• Candidates (candidates.rs): Hydra candidates are built only on a metric with MetricFacts::hydra_dataset, which holds the dataset and the metric's full schema Λ (every schema label, without x). Grids are built only at Λ, since the study measures cost and accuracy at that width: one Hydra candidate per (stream, variant, config, x, y). A Hydra cost row is used only if measured_at.dataset equals the metric's dataset, so rows without a dataset are never candidates. Windows come from RAQEs with ∅ ≠ G_r ⊆ Λ. An edge is eligible for ∅ ≠ G_r ⊆ Λ, with today's window rules and the Hydra accuracy. It is never treated as a roll-up. Λ is a label set, not an ordered list, because the eval datasets fix label order per schema. The old subpopulations ceiling (measured_at_group_count) is removed, since the measured lookup replaces it.
• Docs (rqe_optimizer_candidates.md, rqe_optimizer_cost_model.md, runner module doc): Hydra is held to the worst case (max over covered groups, worst over N and seeds), while per-group sketches read seed-mean curves. Hydra is built only at the full schema because that is what is measured.
• Cost: no formula changes. The existing fixed-size-sketch pricing already matches §3 with I_d = I_r = 1: ingest is λ·(x/y)·c_ins per record, memory is one grid per window, a query makes card(G_r) probes, merges are L/x − 1, and the tracker is priced at Λ as today. Only the module doc is updated.
• MeasuredAt (aqpbm-core): new optional dataset: Option<String> and schema_width: Option<u32> (serde default, skipped if None). PR 6 sets them on Hydra rows. Existing tables load unchanged. MeasuredAt derives no JSON schema, so there is nothing to regenerate.
• Accuracy (saturation.rs): hydra_saturation.csv under the saturation dir is optional. The lookup key is (variant, config params, dataset, grouping as a label set). Raqe::accuracy_covers_share picks the column: None reads err_max; τ ≥ 0.05 reads err_max_cov_0.05; 0.01 ≤ τ < 0.05 reads err_max_cov_0.01; τ < 0.01 reads err_max, which is always the stricter choice. The value is the worst over records and seeds, and an empty cell leaves it unknown. Lossy merges of L/x windows take the worse of the bracketing merge_shards; past the largest the value is unknown. A missing file, dataset or grouping means no accuracy, so the edge is not eligible. read_csv now uses the csv crate because grouping fields are quoted and contain commas. It checks the header for each caller's required columns and fails on a row that is shorter than the header. A config param that isn't name=number is also an error now, where before the row was silently ineligible.
• Runner (autosketch_vs_asap.rs): ASAP, PerQuery and AutoSketch all plan with undeployable families allowed, so every method sees the same non-Hydra families. AutoSketch still skips Hydra itself via one_fixed_size_sketch_for_all_groups (rqe-optimizer: AutoSketch baseline should rank shared fixed-size sketches (HydraKLL) by deployment cost #159). On a schema stream, card(full schema) is the product of the label fan-outs along child_of (the generator's grouping_cardinality: http 10⁴, flows 3·10⁶), capped at the stream's series count so the high-cardinality filter doesn't drop the stream. RQEs carry covers_share. Streams of http map to hydra_http, or hydra_http_latency for quantiles; flows maps to hydra_flows. A stream is one value of one metric, so one dataset per MetricFacts is enough. On workloads with a Hydra dataset, each result reports hydra_rqes.

Tests

Each new test was mutation-checked: 13 mutations, all killed.

• hydra_serves_any_non_empty_subset_of_its_schema covers the subset rule, including non-prefix subsets, ∅, labels outside Λ, and that accuracy still applies.
• hydra_grids_are_built_at_the_full_schema_from_its_dataset_rows checks that only the full schema (without x) gets a grid, with the right windows. It uses two Hydra rows from other datasets (hydra_other, and one with no dataset) and checks that only the hydra_test row is used. It also checks that a metric with no dataset gets no grid.
• flows_and_http_latency_get_a_full_schema_hydra_grid (runner) builds the table from the generator's schemas and adds fixture Hydra rows per dataset. flows gets hydra-hll at {dst_subnet, dst_port, proto}, and http latency gets hydra-kll at all four http labels. flows does not pick up a row measured on hydra_http. flows' card(Λ) is capped at its 1e6 series, and the stream is kept.
• a_malformed_hydra_csv_is_an_error checks three cases: a missing column, a short row, and a non-numeric config param.
• The aqpbm-core cost-row serde test checks that a row with dataset and schema_width serializes and round-trips, and that both fields are left out when None.
• hydra_reads_the_column_its_coverage_selects covers coverage column selection.
• hydra_takes_the_worst_over_records_and_seeds covers worst over N and seeds, and that an empty cell makes the value unknown.
• a_lossy_hydra_merge_reads_the_worse_bracketing_shard_count covers merges 2→max(1,4), 8→16, 32→unknown, and that HLL reads shards = 1.
• without_a_hydra_measurement_no_hydra_deployment_has_accuracy covers a missing CSV, dataset or grouping.
• a_hydra_grid_beats_per_group_hll_on_memory_only_when_accurate is a tiny end-to-end MILP from an inline CSV. Hydra wins on memory at error 0.01 and HLL wins at 0.2.
• The runner tests check the dataset map, that covers_share is threaded through, and hydra_rqes = 0 without the CSV.

Evidence

Round 1 (39a4d16):

• cargo fmt --check, cargo clippy --workspace --all-targets -D warnings: clean. cargo test --workspace: all pass. cargo test -p rqe-optimizer --example autosketch_vs_asap: 8 pass.
• Each new rule was mutated and the tests caught every mutation: without the dataset check, without the series cap, with a lenient config parse, and without the short-row check.
• I regenerated tables with the base branch's generator; they are byte-identical to this branch's. I ran base (561ee86) and this branch with --runs 1 against the committed saturation dir, which has no Hydra CSV or Hydra rows. With *_secs timings stripped, classic shared1 is identical, including AutoSketch now that it allows undeployable families. All shared1 is identical apart from the added "hydra_rqes": 0. Objectives: classic asap 3.31634 / autosketch 3410.17 / perquery 9.553133; all asap 5.052212 / autosketch 3417.90 / perquery 11.755537.

Round 0:

• cargo fmt --check, cargo clippy -p rqe-optimizer --all-targets -D warnings: clean.
• cargo test -p rqe-optimizer: lib 111 (base 104, all 104 kept), integration 2, runner 7 (base 6).
• Base (561ee86) and this branch were run on freshly generated tables (--runs 1) with the committed saturation dir, which has no Hydra CSV and no Hydra cost rows. Classic shared1 is identical once *_secs timings are stripped. All shared1 is identical apart from the added "hydra_rqes": 0. Objectives match: classic asap 3.31634 / perquery 9.553133 (cpu); all asap 5.052212.
• Wiring smoke test, not a result: fixture Hydra cost rows and a fixture CSV on the all table. Hydra candidates appear (793 → 843) and are eligible with the expected column (t14 τ=0.01 → cov_0.01, t15 τ=0.05 → cov_0.05). With the fixture's 45× per-record insert they are not chosen.

Notes

• PR 6's Hydra rows are not here yet. Until they land, no Hydra candidate is eligible in the eval.
• Review round 2 (902cd73): validate_facts rejects a Hydra schema outside the labels or without a nonzero cardinality; the Hydra accuracy fold skips empty cells (unknown only if all are empty); read_csv rejects rows longer than the header. Tests for each, checked by mutation.

Roll-up ablation (b0319d1)

• Runner: new method asap-norollup ("ASAP (no roll-ups)"). It is ASAP's MILP with the same candidates, accuracy, frontier sweep and SLA grid, but allowed keeps each RQE to deployments at its own grouping or Hydra grids. Hydra is not a roll-up. Every result (all methods) gets rolled_up_rqes: the RQEs served by a finer non-Hydra deployment. Sanity checks now cover asap ≤ asap-norollup ≤ perquery, both unbounded and at each SLA, with the existing 1e-6 tolerance. The frontiers use different bounds per method, so they are not compared point by point. No rqe-optimizer/src changes.

• Plots: both scripts draw the ablation as a dashed violet line with ▲ markers. fig_rollup_ablation.png (synthetic script) shows grouped bars of ASAP vs. ASAP (no roll-ups) unbounded cost per workload, in CPU and Fargate panels, with absolute values labeled. summary_synthetic.md and summary_sla.md gain the method's rows and a roll-up section: RQEs rolled up, deployments with/without, cost with/without, and saving %. Results without the method still plot.

• Size: run_autosketch_vs_asap_synthetic.sh passes the existing --no-chosen. rolled_up_rqes is kept either way.

• Tests: forbidding_roll_ups_costs_more_only_where_one_applies covers two cases. With {region} and {region, service}, ASAP rolls up 1 RQE on 1 deployment and is strictly cheaper than no-roll-ups, which has 0 rolled up and 2 deployments. With {region} and {service}, the two costs are equal and nothing is rolled up. The Hydra-grid test checks that a grid finer than its RQE is not a roll-up. Mutation-checked: dropping the Hydra exemption, or never treating a deployment as a roll-up, fails these tests.

• Evidence: fmt, clippy -p rqe-optimizer --all-targets --examples -D warnings, cargo test -p rqe-optimizer (113+2) and the example tests (10) all pass. Smoke run (not committed): fresh --synthetic tables, --runs 1, the 2026-10-08 saturation dir, 0 sanity violations. Unbounded, ASAP vs. no roll-ups:

  • mixed + multi-grouping: cpu 3.53 vs 3.65 (42 RQEs rolled up, 14 vs 30 deployments); fargate 0.233 vs 0.245.
  • mixed + multi-grouping, r = 8: cpu 3.63 vs 3.75.
  • mixed + multi-grouping, m = 8: cpu 28.3 vs 29.2.
  • mixed (classic, every variant): equal, with 0 rolled up.

  The committed tables predate schemas, so on them the two methods are equal. The committed results were not regenerated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis