docs(planner): plan the AutoSketch vs. ASAPQuery planner evaluation (§6.3)

[zzylol]

Before this PR

The §6.3 comparison against AutoSketch had no written plan. The only AutoSketch implementation was an Algorithm 4 adaptation in ASAPQuery-backend (#545 protocol, #547 runner). It is execution-based, CMS-only, and not connected to the RQE MILP that sketch-bench #129 added.

After this PR

docs/evaluation/autosketch-vs-planner.md defines the planner-level comparison, run offline on sketch-bench's rqe-optimizer from sketch-bench measurements.

• Methods (§3):
  • ASAP: the MILP over the whole batch of RQEs.
  • AutoSketch-Adapted: Algorithm 4 once per RQE; memory objective, accuracy only, one sliding sketch per query (x = S, y = T), no sharing.
  • PerQuery-CostAware: ASAP's optimizer per RQE, under the same SLA, no sharing.
• Cost (§4): CPU and retained memory from a 24 h, 1 s CPU timeline; disk excluded.
  • Model A, usage-based: a · avg vCPU + b · GiB, with a and b fit to c7i/m7i/r7i prices.
  • Model B, peak-provisioned: instances = max(peak vCPU / vCPU, GiB / GiB) × price, per family. It is solved for both aligned and staggered evaluation starts.
• Constraints (§5):
  • Accuracy uses per-capability strictness levels; the default matches traces: ARE 0.05, rank error 0.01, top-k precision 0.95.
  • Latency is one absolute SLA for every RQE, swept from 0.01 ms to no limit.
• Accuracy lookup (§6): ASAP reads values past N_sat and KLL/top-k merge curves (Change SimpleStore to have a better indexing for time and label lookups #130, Benchmarking ASAP with Elastic #131, Add experiment scripts to test throughput and cost of ingest path #140), because it merges smaller-window sketches. AutoSketch never merges, so following the paper it reads the measured value at its window size.
• Workloads (§6):
  • synthetic, main figure: 10 PromQL templates and a data model with C groups and s series per group. A workload grid over query mix, replicas, window set, interval, C, s, θ/a, strictness and SLA.
  • traces, appendix: Alibaba 2022, Google 2011, BOOM, with worst-case parameters fit over each whole trace.
• Metrics and figures (§7):
  • $/hour under each model;
  • CPU-seconds, peak CPU and GiB;
  • estimated latency and SLA violations;
  • deployments;
  • planning time: AutoSketch's search plus a benchmark-time lower bound and a paper-rate estimate; ASAP's one-time profiling time reported separately.
• PRs (§8): sketch-bench Change SimpleStore to have a better indexing for time and label lookups #130, Benchmarking ASAP with Elastic #131, Change ArroyoSketch UDF sketches to use sketchlib-rust #137, [Discussion] How to customize binaries for different use cases (like selective compilation with a config as OpenTelemetry collector) #135 and Demonstrating E2E performance after precomputation engine and storage engine are optimized #136 are merged. Test QueryEngineRust binary in experiment scripts after Cargo workspace changes #138, profile existing simple store #139, Add experiment scripts to test throughput and cost of ingest path #140 and PrometheusClient sends a request at /api/v1/status/runtimeinfo to QueryEngineRust, but VictoriaMetrics does not support this end point #141 are open, with experiments running.
• Decisions (§9) follow the AutoSketch paper (§3.2, §5.2, Exp#9): AutoSketch configures once per RQE before deployment and is never re-planned per evaluation.

Validation

Documentation only. Factual claims are checked against sketch-bench (rqe-optimizer at #141's head, the saturation data) and ASAPQuery-backend autosketch_comparison.rs.

🤖 Generated with Claude Code