rqe-optimizer: per-phase cost model, workload facts, weighted AUCCost objective

[milindsrivastava1997]

Summary

Reworks rqe-optimizer's cost model and MILP objective. Covers #143 S3, S7, S2 (plus metric/grouping_labels from S1) and half of S6.

• Objective: milp::minimize(..., Objective::AUCCost { w_cpu, w_mem }) minimizes w_cpu · CPU + w_mem · memory_GiB, default (1, 0). minimize_tco, minimize_cost, MachineFamily and the EC2 pricing snapshot are gone.
• Per-phase costs: ingest, merge, query and storage, each with CPU and memory (table below).
• Workload facts (S2): MetricFacts { labels, scrape_interval_secs, cardinality } replaces LabelSetInfo. λ is derived from it.
• Retained memory (S7): closed windows are (L − x)/y + 1.
• Candidates (S6, half): x and y must be multiples of the scrape interval.

Also: analytical_cost_model.rs (formerly objectives.rs) now holds the one copy of the cost formulas, which score() and the MILP share (they were duplicated before), and candidate dominance compares ingest memory and closed-window storage.

Not run end to end: small_problem --milp needs a fresh scripts/export_rqe_optimizer_costs.sh run. The older local table has no exact-increase rows, so the rate RQEs are unservable with it. --candidates-only runs.

Cost model

Deployment with grouping labels G, window x, slide y. RQE with lookback L, interval T. Measured per-instance costs m, c_ins, c_mrg, c_qry.

Phase	CPU (CPU-sec/sec)	Memory (bytes)
Ingest	λ · (x/y) · c_ins	card(G) · m · x/y (open windows)
Merge	card(G) · (L/x − 1) · c_mrg / T	card(G) · m (one accumulator per group), 0 when L == x
Query	card(G) · c_qry / T	card(G) · 8 B; TopK card(G) · 32 · 16 B
Storage	0	card(G) · m · ((max L − x)/y + 1) (closed windows)

λ = cardinality[metric.labels] / scrape_interval_secs.

$$\min\ w_{cpu} \cdot \text{CPU} + w_{mem} \cdot \text{Memory}_{GiB}$$ $$\text{CPU} = \sum_d \text{ingest}^{cpu}_d\,u_d + \sum_{(r,d)\in E}\left(\text{merge}^{cpu}_{rd} + \text{query}^{cpu}_{rd}\right) z_{rd}$$ $$\text{Memory} = \sum_d \left(\text{ingest}^{mem}_d\,u_d + \text{storage}^{mem}_d\right) + \sum_{(r,d)\in E}\left(\text{merge}^{mem}_{rd} + \text{query}^{mem}_{rd}\right) z_{rd}$$ $$\text{storage}^{mem}_d \ge \text{card}(G_d)\, m_d \left(\tfrac{L_r - x_d}{y_d} + 1\right) z_{rd} \quad \forall (r,d)\in E$$

Shared constraints: $\sum_d z_{rd} = 1$, $z_{rd} \le u_d$, $u_d \le \sum_r z_{rd}$. Latency bounds exclude pairs, so $z_{rd} = 0$ when $\text{lat}_{rd}$ exceeds the bound.

CPU is the mean, i.e. the area under the curve. Plans are sized for average load, not bursts.

Design decisions

Settled in a grilling session. Each entry gives the question and the answer as agreed.

Before grilling

• Scope: "Let's just tackle them all here" in rqe-optimizer: per-phase cost model, workload facts, weighted AUCCost objective #145, rather than splitting into separate issues.
• Ingest memory: open windows only. Closed windows go in a separate Storage phase with CPU = 0.
• Merge memory: card(G) · m, one accumulator per group, with all groups merged at once. The windows folded in are live objects already counted in storage, so they are not counted again. A direct query (L == x) merges nothing and has no merge memory.
• Query labels: a query is served only by a deployment with exactly the same G for now. No regrouping of finer deployments.
• Cost scaling with keys: costs should ideally grow with keys per instance. Superseded by Q15.
• Objective: remove Tco, and rename Cost to AUCCost.

Q1. Do S1 and S2 come into #145? S2 in full, plus only metric and grouping_labels from S1. The Raqe rename, milliseconds, spatial_filter and the latency SLA on the query stay in S1.

Q2. How are costs scaled with keys per instance? Per-family laws with measured_keys. Superseded by Q15.

Q3. Which parts of S4b come in? Shape (PerGroup/Shared) × law (Fixed/PerKey) in. Mergeable/subtractable, the key tracker and HydraKLL stay in S4b. Shape × law later dropped by Q15.

Q4. Formula for storage? Adopt the S7 formula: closed windows (L − x)/y + 1, for the longest L the deployment serves.

Q5 / Q6. How do the phases feed the objective, and who provides the weights? Simplify the objective to w_cpu · CPU + w_mem · memory, starting at (1, 0). Memory is option C: the sum of all memory terms, as if every query evaluates at once. The weights are machine-family constants named w_cpu and w_mem.

Q7. How does #145 relate to #141? Independently. #141 (CostModel::Usage/Peak, minimize_model_cost) overlaps with this and will conflict in milp.rs.

Q8. What happens to MachineFamily's vcpu, memory_gib and usd_per_hour? Drop them; no $/h reporting.

Q9. Where do the weights come from? Option A: delete the EC2 pricing infrastructure (snapshot, fetch_ec2_pricing.py, from_pricing_json, --machine-family). The weights are given directly.

Q10. Is MachineFamily still needed? No, delete it. The weights live in Objective::AUCCost { w_cpu, w_mem }.

Q11. Should merge and query CPU scale with keys for PerKey families? Agreed, scale merge and query too. Superseded by Q15.

Q12. Is K (key labels) needed, and is the top-k heap modeled? Skip K, since no in-scope family is PerKey with non-empty K. Treat top-k as fixed at the benchmark's CMS_HEAP_TOP_K = 32: the heap is excluded from memory and included in CPU.

Q13. What is the S2 data model? Final shape:

• MetricFacts { labels, scrape_interval_secs, cardinality: BTreeMap<LabelSet, u64> }, keyed by metric name.
• λ = cardinality[labels] / scrape_interval, derived only, no override. This reverses Tracking: make rqe-optimizer the solver for ASAPQuery's planner #143's "arrival rate is an explicit per-metric input".
• Rqe and Deployment carry metric and grouping_labels. Eligibility and sharing match on (capability, metric, grouping_labels).
• x and y must be multiples of the scrape interval (half of S6).
• Validation happens up front and reports every problem at once:
  • a metric is missing
  • G ⊄ labels
  • a cardinality entry is missing for G or for labels
  • card = 0 or scrape = 0
  • card(G) > card(labels)

Q14. What do the scored output and the Pareto front look like?

• Keep the Pareto front, with the vector (cpu, memory, {latency}).
• Objectives becomes per phase: ingest, merge, query and storage, each a PhaseCost { cpu_secs_per_sec, memory_bytes }, plus query_latency_secs. It has cpu() and memory_bytes() totals. Later renamed PlanCost, in analytical_cost_model.rs (formerly objectives.rs), since it is a plan's cost and Objective already names what the MILP minimizes.
• Remove peak_query_memory_bytes, retained_memory_bytes and tco_cpu_secs_per_sec.
• w_mem is per GiB.

Q15. Drop shape × law and measured_keys? Yes. The export already divides the exact accumulators' memory and merge by groups_per_instance (aqpbm-core/src/atomic_costs.rs), so card(G) × m is already correct for every in-scope family. Defer both to #142, where HydraKLL (Shared + Fixed) first needs them.

Q16. How many bytes per output entry for query memory? Constants: 8 B per value, and 16 B per top-k (key, count) entry, with 32 entries per group.

Q17. Final scope

In #145:

• S3 (also remove the peak-memory bound)
• the weighted AUCCost objective
• the per-phase costs
• S7
• S2, plus metric and grouping_labels
• the S6 scrape-multiples filter
• per-phase Objectives, and the Pareto front kept
• docs and tests

Not in #145:

• the rest of S1
• S4b and rqe-optimizer: support shared multi-subpopulation sketches (HydraKLL) in the cost model #142
• S5
• the S6 allowed families
• S8
• K and top-k k scaling
• reconciling with Synthetic workload results: reproduction steps and ASAP's profiling time #141

Overlap

• Synthetic workload results: reproduction steps and ASAP's profiling time #141 also changes the objective and milp.rs. Whichever PR merges second has to reconcile the two.
• Tracking: make rqe-optimizer the solver for ASAPQuery's planner #143's decision on λ is reversed here; a comment on Tracking: make rqe-optimizer the solver for ASAPQuery's planner #143 explains it.

Test plan

• MILP matches brute force on the new objective at weights (1, 0), (0, 1) and (1, 4), including µs-scale costs
• w_mem flips the choice between a CPU-heavy and a memory-heavy deployment
• a unit test for each phase formula, the S7 closed-window count, a direct query (L == x), and storage sized by the longest lookback of a shared deployment
• validation reports each kind of bad input at once
• scrape-interval filter on candidates
• cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings (pre-commit)
• small_problem --milp on a fresh cost export

🤖 Generated with Claude Code

[zzylol]

Given that you model the window sharing in the doc, can you model the sharing across queries in the doc as well, which will guide the MILP search space?

[zzylol]

What does pareto in milp refer to?

[milindsrivastava1997]

Good catch. MILP doesn't use the Pareto vector. This was added earlier for manual exploration of different plans that are the Pareto frontier of "CPU, memory, latency". Dead code right now.