eval: AutoSketch vs. ASAP with Hydra candidates and a roll-up ablation (§6.3)

[zzylol]

Results of the §6.3 synthetic eval (ProjectASAP/ASAPQuery#777) with Hydra candidates and a roll-up ablation. Everything is under rqe-optimizer/results/autosketch-vs-asap-hydra/: figures, result JSONs, summaries, inputs, and a README with the analysis.

Base. The base is hydra/eval-integration, which merges the open stack #195 → #199 → #200 → #202 (with #197 and #198) and #196 → #201. Retarget to main once those merge. The accuracy curves come from #138's study, committed in #194.

Findings

• Roll-ups (rqe-optimizer: let mergeable finer-group deployments serve coarser RAQEs #190) help ASAP only a little. On the multi-grouping set, 40% of RQEs are served from a finer deployment (42 of 104, and the same share at r=8, m=8, m=16), and ASAP's deployments halve (30 → 14; 480 → 224 at m=16). Cost drops by only 3.3–3.6% under CPU weights and 4.4–4.8% under Fargate, at every frontier point. The classic set has one grouping per stream, so roll-ups never apply there. ASAP's ~3× advantage over PerQuery comes from sharing, not from roll-ups.
• Hydra is never chosen. It is eligible for many RQEs, but neither ASAP nor PerQuery picks it in any workload, weight setting, bound or SLA. Its insert fans out to every label subset: 1.9–5.1 µs per record, against 3.85 ns for a per-group HLL. A check on real traces (Alibaba 2022, Google 2011) agreed. Per #777's plan, Hydra is therefore not implemented in ASAPQuery.
• Overall: 0 RQEs dropped and 0 sanity violations in all 8 workloads. ASAP < ASAP without roll-ups < PerQuery << AutoSketch.

Workload	Weights	ASAP	ASAP (no roll-ups)	PerQuery	AutoSketch
mixed (classic), 50 RQEs	CPU (vCPU)	3.54	3.54	10.5	3.72e3
mixed + multi-grouping, 104	CPU	3.79	3.94	11.3	3.72e3
mixed + multi-grouping, 104	Fargate ($/h)	0.245	0.257	0.799	176
mixed + multi-grouping, m=16, 1664	CPU	60.7	63.0	181	5.96e4

Figures

• fig_frontier.png: cost vs. query latency.
• fig_cost_vs_sla.png: cost vs. batch-latency SLA.
• fig_planning_time.png: planning time.
• fig_rollup_ablation.png: ASAP with vs. without roll-ups.

The README has every table, the method definitions, the inputs, the reproduce commands and the caveats.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis

[zzylol]

Added f8cecf9: template 16 is now p99 latency GROUP BY CUBE(region, service, endpoint) (all 7 groupings), and a new multigroup set runs the multi-grouping templates without the classic ones. Roll-ups save 60% on multi-grouping (0.551 → 0.221 vCPU, 26 → 5 deployments) and 8% on mixed + multi-grouping. Classic is unchanged, and Hydra is still never chosen. README and ASAPQuery#777 §11 (ba6b4e3) are updated.

🤖 Generated with Claude Code