eval: AutoSketch vs. ASAP on the synthetic mixed set (cost by use; frontier and SLA versions)

[zzylol]

What

The AutoSketch vs. ASAP planner evaluation (ProjectASAP/ASAPQuery#777) on the synthetic mixed template set. It contains everything behind its three figures: code, scripts, inputs (including the raw approxbench records), results and figures.

Based on main, which has #188 (cost by use and batch latency).

Figures

• Version 1, no SLA: each method's cost–latency frontier, in rqe-optimizer/results/autosketch-vs-asap-synthetic/fig_frontier.png.
• Version 2, batch latency SLA: cost vs. SLA, in rqe-optimizer/results/autosketch-vs-asap-synthetic-sla/fig_cost_vs_sla.png.
• Planning time vs. RQEs: AutoSketch shown with its measured benchmark time, in rqe-optimizer/results/autosketch-vs-asap-synthetic/fig_planning_time.png.

Setup

• Methods:
  • ASAP: minimize_usage_cost with sharing.
  • PerQuery-CostAware: the same MILP, each RQE limited to its own candidates, so nothing is shared.
  • AutoSketch-Adapted: Algorithm 4 per RQE, its plan priced the same way.
• Cost by use (rqe-optimizer: MILP on cost by use, with batch latency (reported, or as an SLA) #188), two weight settings: CPU only (vCPU), and Fargate prices ($/hour). Costs are absolute.
• Latency: the plan's batch latency.
  • Version 1 sweeps 12 latency bounds plus AutoSketch's latency.
  • Version 2 uses SLAs of {100, 300, 1000, 3000, 10000} ms.
• Workloads: the mixed set, 50 RQEs; shared replicas r = 8, 92 RQEs; metrics m ∈ {8, 16}, 400 and 800 RQEs. Accuracy target p95.
• AutoSketch's planning time is its search plus its measured benchmark: each distinct probed (config, data shape) runs approxbench's accuracy benchmark at 1e8 items, including the exact baseline and scoring, serially on an idle machine. It is charged once per metric. 60 s per probe appears only as a reference.

Results

All costs CPU-only, in vCPU. AutoSketch's latency is 1011 ms everywhere.

Workload	ASAP cheapest (latency)	PerQuery cheapest (latency)	AutoSketch	ASAP at 1011 ms	ASAP at 100 ms SLA
mixed (50)	3.32 (12.1 s)	9.55 (3.84 s)	3410	5.12	34.5
r = 8 (92)	3.39 (2.94 s)	16.7 (3.84 s)	4089	5.15	34.5
m = 8 (400)	26.5 (12.1 s)	76.4 (3.84 s)	27,280	41.0	276
m = 16 (800)	53.1 (12.1 s)	153 (3.84 s)	54,560	81.9	553

Planning time at 50 / 400 / 800 RQEs:

• ASAP: 0.76 / 6.8 / 14.7 s.
• AutoSketch: 212 / 1697 / 3394 s, almost all of it measured benchmark.

No RQE was dropped and every sanity check passed: ASAP ≤ PerQuery, ASAP ≤ AutoSketch, frontiers monotone, every SLA met.

Contents

• Code: rqe-optimizer/examples/autosketch_vs_asap.rs, the runner.
• Scripts:
  • export_autosketch_eval_table.py and its test (synthetic tables);
  • run_autosketch_vs_asap_synthetic.sh;
  • autosketch_benchmark_time.py;
  • plot_autosketch_vs_asap_synthetic.py;
  • plot_autosketch_vs_asap_sla.py.
• Inputs: rqe-optimizer/results/autosketch-vs-asap-inputs/.
  • saturation/: the sketch-bench study on asap_sketchlib 0.3.0. It has the cost table (sketches and exact aggregations) and the accuracy curves; the runner reads it as --saturation-dir.
  • tables/: the four workload tables and plan.tsv.
  • raw/: the raw approxbench records behind saturation/, gzipped (920 KB), with a README mapping each file to its machine, arguments and run time. Rerunning them is 0.77 h of machine time in all, run in parallel on 5 machines (about 20–30 minutes of wall time; the serial cost table alone is 17 minutes).
• Results:
  • results/autosketch-vs-asap-synthetic/: version 1.
  • results/autosketch-vs-asap-synthetic-sla/: version 2.
  • Each has results JSON without per-RQE choices, a summary, a README, AutoSketch's benchmark times, and its figures.

Rerunning one workload from the committed inputs reproduces the committed results exactly. The READMEs give the steps.

Not here

The Alibaba and Google traces (with 6h and 24h windows, ASAPQuery #812) are to be rerun on this model in a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis