Repository navigation
rqe-optimizer: support shared multi-subpopulation sketches (HydraKLL) in the cost model #142
Description
Problem
rqe-optimizer treats every capability as one sketch per group. A deployment for label set L is card(L) independent instances. Memory and ingest are priced as card(L) × per-instance cost, using LabelSetInfo.cardinality, and accuracy is one per-instance number.
HydraKLL (hydra-kll, SubpopQuantile in sketch-bench) works differently. It is one sketch that answers a quantile for every group. Its memory doesn't scale with card(L), and its accuracy depends on how many groups share it. Adding "hydra-kll" to Capability::Quantile.families() would misprice it in both directions.
What's needed
- A way for a cost entry to say it is a shared instance: one per deployment, not
card(L). - Memory and ingest priced from that (
active_instance_count × per-instance cost, with no× card(L)). - Accuracy measured at the group count the RQE's label set actually has, or a curve over group count. The current single
query_accuracyvalue is taken at whatever group count the bench spec happened to have. - Then add
hydra-kllas a Quantile candidate. The same would lethydra-cms,hydra-hllandhydra-univmon-*in.
Related
The exact *_over_time accumulators (exact-sum, exact-max, exact-increase) are now multi-subpopulation, mirroring ASAPQuery's Multiple*Accumulator. One instance holds every group, so their measured mem_bytes_per_instance is for the whole label set at the bench spec's group count (~9.9k groups in hydra_columns.yaml). The optimizer still multiplies it by card(L). The same "shared instance" fix should apply to them.
🤖 Generated with Claude Code
Activity
milindsrivastava1997 commentedon Oct 5, 2026
Update: the exact accumulators part under "Related" is now handled differently.
atomic_costs::reduce_onedivides memory and merge cost by thegroups_per_instancethe comparator reports. That makes their entries per-group, so the optimizer'scard(group-by) ×is already correct for them. This issue is now only about the Hydra sketches, whose accuracy (not just cost) depends on how many groups share one instance.milindsrivastava1997 commentedon Oct 6, 2026
Correction to the "Related" section above: the exact accumulators are not overpriced by
card(L)today.aqpbm-core/src/atomic_costs.rsdivides their measuredmem_bytes_per_instanceandmerge_cpu_secsbygroups_per_instance(the bench spec's group count, recorded inquery_accuracy). Query cost is already per group. So the rows are per-group costs, and the optimizer'scard(G) × mprices each group once, which is correct.What this issue still needs is unchanged for Shared + Fixed families like HydraKLL: one fixed-size instance per deployment, which is not ×
card(G), and accuracy that depends on group count. Per #145's scope (Q15 there), that's where the per-family "shape" property (PerGroup / Shared) belongs.refactor(bench): one measure helper; HydraKLL on disjoint label alpha…
feat(bench): exact-delta-set row and HydraKLL in the rqe-optimizer co…
feat(rqe-optimizer): family properties, shared fixed-size sketches an…
zzylol commentedon Oct 10, 2026
Superseded by the Hydra admission design in #193 (
docs/rqe_optimizer_hydra.md).one_fixed_size_sketch_for_all_groups,I_d = 1.I_d = I_r = 1, a probe cost per answered group, and the key tracker as a part.Closing in favor of #193 (decided by @zzylol).
🤖 Generated with Claude Code