rqe cost export: sweep items per instance and re-export (follow-up to #147) #157

Goal

Finish #147's last item and put its merged fixes into a real cost table: sweep items per instance in scripts/export_rqe_optimizer_costs.sh, and let rqe-optimizer read each row's costs and accuracy at the deployment's own instance size instead of the benchmark's.

Why

#145's cost model applies each row to instances of λ × x / card(G) items per window. Real windows are often far smaller than the benchmark's 1,000,000 values (200,000 rows for the column-spec runs). Since #155 every row records measured_at.items_per_instance, so the gap is visible, but there is still only one size to read from.

From #147's analysis, the size affects:

Work

• Re-export once with what is merged (rqe cost export: count top-k heaps, time top-k queries above the clock floor, add exact hash-table overhead #151, rqe-optimizer: size DDSketch memory from the metric's value range #152, rqe-optimizer: check accuracy at the query's merge count #154, atomic-costs: record the measurement conditions on each cost-table row #155) into a new directory on an idle machine (clnode109): about 1.5 h. This replaces out_10_6_26_1342 (18 rows, pre-feat(bench): exact-delta-set row and HydraKLL in the rqe-optimizer cost export #149, pre-rqe cost export: count top-k heaps, time top-k queries above the clock floor, add exact hash-table overhead #151). Check measured_at and merge_accuracy on every row, and the top-k and exact rows' new memory.
• Size grid in the export: measure cost and accuracy (single-instance and merged) at several --size values, e.g. 1e4, 1e5, 1e6. Each row keys on its measured_at.items_per_instance. Expect about 3× the export's runtime.
• Optimizer reads at the deployment's size: pick or interpolate each row's costs and accuracy at λ × x / card(G) (merge accumulator: λ × L / card(G), as rqe-optimizer: size DDSketch memory from the metric's value range #152 does for DDSketch memory). Rows with one size keep today's behavior.
• Design first: grill the grid, the interpolation rule (nearest, log-linear, conservative bracket as in rqe-optimizer: check accuracy at the query's merge count #154) and what happens outside the grid before writing code.

Related

• Tracking: make the rqe-optimizer cost export fit the per-phase cost model #147: the tracking issue this continues (closed with the other items done).
• rqe-optimizer: per-phase cost model, workload facts, weighted AUCCost objective #145: the per-phase cost model.
• rqe cost export: count top-k heaps, time top-k queries above the clock floor, add exact hash-table overhead #151, rqe-optimizer: size DDSketch memory from the metric's value range #152, rqe-optimizer: check accuracy at the query's merge count #154, atomic-costs: record the measurement conditions on each cost-table row #155: the merged Tracking: make the rqe-optimizer cost export fit the per-phase cost model #147 PRs.

🤖 Generated with Claude Code