Saturation curves at label cardinalities 1e1–1e6 for the synthetic planner workload

[zzylol]

Draft: accuracy results are in; cost results are pending (62 top-k points running in parallel, plus a serial-vs-parallel consistency check).

Before this PR

The saturation grid (#130) and merged-accuracy runs (#131) measured K ∈ {1e3, 1e5, 1e7}. The synthetic PromQL workload of the AutoSketch vs. ASAP planner evaluation (ProjectASAP/ASAPQuery#777, plan §6) varies card(label_0) over 1e1–1e6, so its accuracy lookups at 10, 100, 1e4 and 1e6 had no measured curves.

After this PR

docs/figures/saturation/synthetic_cardinalities/ adds measured frequency and top-k curves at K ∈ {10, 100, 1e4, 1e6} (θ ∈ {0, 0.5, 1, 1.5, 2}, N up to 1e8, 3 seeds), top-k accuracy after merging m ∈ {4, 16, 64} shards (N ≤ 1e7), the N_sat table per (sketch, config, θ, K), and the summary CSVs (2.1 MB). Quantile curves don't depend on K and are unchanged.

Findings:

• K ≤ 100 saturates early: every point reaches N_sat between 1e3 and 6e4.
• Large K with low skew may never saturate: at K = 1e4 and 1e6 with θ ≤ 0.5, many points are still moving at N = 1e8 (35 of 240 points at K ≥ 1e4). Their N_sat is a lower bound.
• Top-k merging loses precision at large K: merged/single precision@k at N = 1e7 drops to 0.50 (rows=3 cols=16384, θ = 0, K = 1e4) and to 0.82–0.90 at K = 1e6.

Scripts:

• summarize_synthetic_curves.py: the README's tables.
• complete_saturation_costs.py: finishes an interrupted cost phase (the cost phase rewrites its output and can't resume). It runs the same approxbench command, --jobs at a time.
• check_parallel_costs.py: re-measures serially measured points as one parallel batch and reports the parallel/serial CPU ratio, so parallel cost results can be checked before use.

Cost phase (pending)

The serial cost phase measured 178 of 240 rows=3 points (all CMS and CountSketch, 18 top-k), then was stopped: the remaining top-k points took 10–20 min each. The other 62 are running 12-way in parallel. Then 12 serial points (4 per sketch, every K and θ) will be re-measured as one 12-way batch and compared. If insert/merge/query CPU differs by more than about 10%, the 62 will be re-run serially. The README will list which points were measured in parallel. The PR leaves draft once that's done.

Validation

• Accuracy phase on CloudLab clnode109 (Xeon E5-2683 v3, 56 CPUs), both runs exited 0. Before moving to the remote node, two configs re-run at K = 1e3/1e5, θ = 1.0 matched the existing Add saturation study: error vs N, N_saturation and Pareto value data #130 curves.
• README counts recomputed from saturation.csv: 35/240 unsaturated at K ≥ 1e4; max N_sat 5.6e4 and none unsaturated at K ≤ 100.
• Scripts import and compile; the summary script regenerates the README tables from the committed CSVs.

Links: ProjectASAP/ASAPQuery#777, #130, #131.

Cost phase (50f2acf)

All 240 rows=3 points now have cost in saturation.csv, every one measured serially on clnode109:

• 178 points (all CMS, all CountSketch, 18 top-k) at N = 1e8 by study_saturation.py --phase cost.
• 62 top-k points at N = 1e7 by complete_saturation_costs.py --jobs 1 --n-max 1e7 --normalize-n 1e8. The serial 1e8 phase was stopped: those points take 30–50 min each, about 25–50 h in total.
• 1e7 validation (scripts/check_cost_size.py): five points with serial 1e8 records were re-measured at 1e7.
  • The three top-k points are within 5% on insert and merge (merge 1.051 at one point, accepted as measured) and within 10% on the query phase.
  • CMS merge (7.3×, a µs-scale merge) and the CountSketch query phase at K = 1e6 (1.5×) fail, but those sketches are all measured at 1e8.
  • Full table in the README.
• Insert normalization: the 1e7 points' insert CPU is written scaled to N = 1e8, the N readers divide by, so insert CPU per item = each record's CPU / its own N.
• Discarded: the first 12-way parallel measurement of the 62 points. Its consistency check showed CPU shifts above 10% (insert 1.17–1.20×, one merge at 0.04×). Table in the README; the raw records are kept but unused.

Use in the evaluation (#141, ce4b885). The evaluation also reads CPU per query from this PR's raw cost records (query-phase CPU over query count). The 62 top-k points at N = 1e7 supply it too. Their query-phase ratio to 1e8 was 0.92–1.07 in the validation.

🤖 Generated with Claude Code

[zzylol]

Closing. The synthetic evaluation no longer needs these curves:

• Evaluate the synthetic PromQL workload of ASAPQuery#777 #139 fixes the synthetic data at K = 1e4 and θ = 1.1. rqe-optimizer: one study for sketch cost and accuracy (#174) #178 adds that shape to the default curve grid, so every config gets a curve there.
• These curves and costs also predate rqe cost export: count top-k heaps, time top-k queries above the clock floor, add exact hash-table overhead #151 (top-k heap wrapper) and Report DDSketch relative value error #164/study_saturation: record DDSketch mean relative value error #176 (DD metric), so they would be regenerated anyway. The full grid, including merge curves, is being rerun on clnode109 for rqe-optimizer: one study for sketch cost and accuracy (#174) #178.

#141 will be restacked onto the regenerated curves and will drop its references to synthetic_cardinalities/.

🤖 Generated with Claude Code