Skip to content

Commit 8132a74

Browse files
feat(planner): optimizer accepts label-set facts and costs by cardinality (#780)
* feat(planner): load label-set facts for the optimizer Replace --dataset/--rho with --label-set-facts: series_count per (metric, spatial filter) and cardinality per (metric, spatial filter, grouping labels). Arrival rate is derived as series_count / scrape interval. The label schema now comes from the workload's metrics: hints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(planner): cost optimizer candidates by label-set cardinality Cardinality now scales cost by how each sketch holds its groups: keyed maps grow per key, fixed-size sketches don't, and grouped queries read one value per output group (one heap read per top-k bucket). Trivial accumulators get an analytical per-key memory estimate, and only their Multiple* forms are proposed. Optimizer configs now use the legacy planner's grouping/aggregated label split, so keyed sketches deploy as one instance rather than one per group, and CMS/HydraKLL deploy their paired DeltaSet key aggregation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(planner): document label-set facts and cardinality costing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(planner): keep top-k in the optimizer example workload Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 0e42b15 commit 8132a74

21 files changed

Lines changed: 1369 additions & 898 deletions
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Label-set facts for optimizer-workload.example.yaml (asap-optimizer-cli --label-set-facts).
2+
series:
3+
- {metric: http_requests_total, spatial_filter: 'env="prod"', series_count: 200}
4+
- {metric: latency_ms, spatial_filter: "", series_count: 50}
5+
groups:
6+
# sum by (job): output labels [job]; also the `topk by (job)` buckets.
7+
- {metric: http_requests_total, spatial_filter: 'env="prod"', grouping_labels: [job], cardinality: 10}
8+
# topk keeps every label: output labels are all of them.
9+
- {metric: http_requests_total, spatial_filter: 'env="prod"', grouping_labels: [env, instance, job], cardinality: 200}
10+
# quantile_over_time keeps every label.
11+
- {metric: latency_ms, spatial_filter: "", grouping_labels: [instance], cardinality: 50}

‎.design_docs/optimizer-v1-implementation-plan.md‎

Lines changed: 82 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,12 @@ The optimizer output type is `OptimizerSolution`, which is then translated into
3535
|---|---|
3636
| `mergeable` | Two instances can be combined → supports Merge query method |
3737
| `subtractable` | Element-wise difference defined → supports Subtract (tumbling only) |
38-
| `subpopulation_aware` | One instance handles multiple label-group keys → N(s,g)=1 |
38+
| `memory_grows_with_keys` | One instance is a keyed map whose memory grows per key (`Multiple*`), vs a fixed-size instance |
39+
40+
How many instances a config has comes from its `grouping_labels`, split the same way as the
41+
legacy planner (`planner/labels.rs::set_subpopulation_labels`): keyed types keep the output
42+
labels in `aggregated_labels` (one instance), per-group types in `grouping_labels` (one instance
43+
per group), and `topk by (L)` gets one heap per value of `L`.
3944

4045
### Query method (not a free variable — derived from ingest_type × W vs range_a × algebra)
4146

@@ -76,7 +81,7 @@ asap-planner-rs/src/optimizer/
7681
├── translator.rs translate(&OptimizerSolution) → (StreamingConfig, InferenceConfig)
7782
├── aqe_extractor.rs Rqe, extract_aqes()
7883
├── pipeline.rs run_all_exact_pipeline(), run_greedy_pipeline()
79-
├── dataset.rs CSV series inventory, schema validation, and N_g profiling [#693]
84+
├── label_set_facts.rs external series counts / group cardinalities → per-AQE facts [#756]
8085
├── sketch_properties.rs algebraic properties per AggregationType [Phase 2a, done]
8186
├── candidate_gen.rs enumerate candidate configs per AQE [Phase 2b, done]
8287
├── cost_model.rs ingest/query cost formulas [Phase 2c, done]
@@ -126,20 +131,23 @@ Each AQE gets its own best config independently (no cross-AQE sharing). MIP shar
126131
#### 2a — `sketch_properties.rs` (done)
127132

128133
```rust
129-
pub struct SketchProperties { pub mergeable: bool, pub subtractable: bool, pub subpopulation_aware: bool }
134+
pub struct SketchProperties { pub mergeable: bool, pub subtractable: bool, pub memory_grows_with_keys: bool }
130135
pub fn sketch_properties(t: AggregationType) -> SketchProperties
131136
```
132137

133138
Implemented values — `CountMinSketchWithHeap` ended up `mergeable=false` (not `true`/unclear as
134139
originally guessed): the heap top-k list doesn't compose across merged/subtracted windows even
135-
though the underlying CMS cells would. Everything else matches the original guess:
136-
- `CountMinSketch`: mergeable=T, subtractable=T, subpopulation_aware=T
137-
- `CountMinSketchWithHeap`: mergeable=F, subtractable=F, subpopulation_aware=T
138-
- `DatasketchesKLL`, `HydraKLL`: mergeable=T, subtractable=F, subpopulation_aware=F (true for HydraKLL too)
139-
- `Sum`/`MultipleSum`: mergeable=T, subtractable=T, subpopulation_aware=F/T respectively
140-
- `HLL`, `SetAggregator`, `DeltaSetAggregator`: mergeable=T, subtractable=F, subpopulation_aware=F
141-
- `MinMax`/`MultipleMinMax`, `Increase`/`MultipleIncrease`: mergeable=T, subtractable=F, subpopulation_aware=F/T
142-
- `SingleSubpopulation`/`MultipleSubpopulation` (legacy wrapper types): all false (unknown, treated conservatively)
140+
though the underlying CMS cells would. `memory_grows_with_keys` is true only for the `Multiple*`
141+
maps:
142+
- `CountMinSketch`: mergeable=T, subtractable=T
143+
- `CountMinSketchWithHeap`: mergeable=F, subtractable=F
144+
- `DatasketchesKLL`, `HydraKLL`: mergeable=T, subtractable=F
145+
- `Sum`/`MultipleSum`: mergeable=T, subtractable=T
146+
- `HLL`, `SetAggregator`, `DeltaSetAggregator`: mergeable=T, subtractable=F
147+
- `MinMax`/`MultipleMinMax`, `Increase`/`MultipleIncrease`: mergeable=T, subtractable=F
148+
149+
The optimizer never proposes single-group `Sum`/`MinMax`/`Increase`: with analytical per-key
150+
memory they cost the same as their `Multiple*` forms (`OPTIMIZER_SKIPPED_AGG_TYPES`).
143151

144152
When a type is both `mergeable` and `subtractable` (e.g. `Sum`, `CountMinSketch`),
145153
`candidate_gen.rs` prefers `Subtract` — it's O(1) regardless of `n`, strictly cheaper than `Merge`'s O(n).
@@ -151,11 +159,18 @@ pub struct CandidateConfig {
151159
pub config: Option<AggregationConfig>, // None = EXACT fallback
152160
pub query_method: QueryMethod,
153161
pub n_windows: u64,
154-
pub label_group_count: u64, // dataset-derived N_g
162+
pub instance_count: u64, // from the config's grouping labels
163+
pub output_group_count: u64, // cardinality of the AQE's output labels
164+
pub key_config: Option<AggregationConfig>, // paired DeltaSet for CMS/HydraKLL
155165
}
156-
pub fn enumerate_candidates(aqe: &Aqe, scrape_interval_secs: u64) -> Vec<CandidateConfig>
166+
pub fn enumerate_candidates_with_facts(aqe: &Aqe, scrape_interval_ms: u64, facts: &ItemFacts) -> Vec<CandidateConfig>
157167
```
158168

169+
CMS and HydraKLL can't list their own keys, so each such candidate carries the same paired
170+
DeltaSetAggregator the legacy planner deploys (`needs_key_aggregation`): same labels, Tumbling
171+
at the value's slide, retaining `⌈range_a / slide⌉` panes. Greedy registers both and the
172+
translator references both in the query config.
173+
159174
Enumeration: `compatible_agg_types(stat)` × param grid (hardcoded small grids per sketch type,
160175
see `CMS_DEPTHS`/`CMS_WIDTHS`/etc. constants — replace with sketch-bench sweep results in Phase 3)
161176
× window candidates × ingest type:
@@ -179,15 +194,23 @@ Label granularity: only proposes configs at the label granularity the AQE itself
179194
pub struct AtomicCosts { mem_bytes_per_instance, insert_cpu_secs, merge_cpu_secs, subtract_cpu_secs,
180195
query_cpu_secs, exact_query_cpu_secs: f64 }
181196
pub struct CostWeights { ingest_mem, ingest_cpu, query_mem, query_cpu: f64 }
182-
pub fn ingest_cost(candidate: &CandidateConfig, rho_g: f64, costs: &AtomicCosts, weights: &CostWeights) -> f64
197+
pub fn ingest_cost(candidate: &CandidateConfig, arrival_rate_hz: f64, costs: &AtomicCosts, weights: &CostWeights) -> f64
183198
pub fn query_cost(a: &Aqe, candidate: &CandidateConfig, costs: &AtomicCosts, weights: &CostWeights) -> f64
184-
pub fn total_cost_rate(a: &Aqe, candidate: &CandidateConfig, rho_g: f64, costs: &AtomicCosts, weights: &CostWeights) -> f64
199+
pub fn total_cost_rate(a: &Aqe, candidate: &CandidateConfig, arrival_rate_hz: f64, costs: &AtomicCosts, weights: &CostWeights) -> f64
185200
```
186201

187-
Implements the design doc's formulas with `N(s,g) = 1` for subpopulation-aware sketches and
188-
the dataset-derived `N_g` for per-label-group sketches. The CSV series inventory and exact
189-
filter profiling are implemented in `dataset.rs` (#693); Prometheus-backed profiling remains
190-
the follow-up in #525.
202+
Group cardinality enters per candidate:
203+
204+
| | Multiplier |
205+
|---|---|
206+
| Memory, merge/subtract CPU | `output_group_count` for keyed maps (`Multiple*`), else `instance_count` |
207+
| Query read CPU | `output_group_count` (one read per output group); top-k: `instance_count` (one heap read each) |
208+
| Insert CPU | arrival rate `λ`, independent of cardinality |
209+
| Paired key tracker | `output_group_count` key entries + one insert per item |
210+
211+
Trivial accumulators (`Sum`/`MinMax`/`Increase` and `Multiple*`) use an analytical per-group
212+
entry, `(n_labels × 4 + value_bytes) × 8/7`: label values dictionary-encoded as `u32` codes
213+
(dictionary amortized, not charged), hashbrown load-factor slack. CPU stays on the stub.
191214
`AtomicCosts` added `exact_query_cpu_secs` (not in the original plan) — without a non-zero cost
192215
for the EXACT fallback's query, `IngestCost=0, QueryCost=0` would make EXACT always win trivially.
193216

@@ -212,18 +235,19 @@ it then, reusing `window_compatible`, `spatial_filter_compatible`, `topk_weighti
212235
#### 2e — `greedy.rs` + `pipeline.rs` + `translator.rs` (done)
213236

214237
```rust
215-
pub fn greedy_assign(aqes: Vec<Aqe>, scrape_interval_secs: u64, rho_g: f64,
216-
costs: &AtomicCosts, weights: &CostWeights) -> OptimizerSolution
238+
pub fn greedy_assign(aqes: Vec<Aqe>, scrape_interval_ms: u64, atomic_cost_table: &AtomicCostTable,
239+
weights: &CostWeights, facts: &[ItemFacts]) -> OptimizerSolution
217240
```
218241

219242
For each AQE independently: `argmin` over `enumerate_candidates(aqe, ...)` by `total_cost_rate`.
220243
No feasibility filter needed (see 2d note) — every candidate from `enumerate_candidates` is valid
221244
for that AQE by construction. Assigns sequential `aggregation_id`s to deployed configs.
222245

223-
`run_greedy_pipeline(config, dataset, scrape_interval_secs, rho_g) -> Result<(StreamingConfig, InferenceConfig), DatasetError>`
224-
added to `pipeline.rs` alongside `run_all_exact_pipeline()`. The dataset is required for the
225-
greedy path and supplies metric schemas plus the `N_g` profile for each AQE. `rho_g` is
226-
currently a single placeholder value applied uniformly to every candidate — see open TODO below.
246+
`run_greedy_pipeline(config, facts, scrape_interval_ms, atomic_cost_table) -> Result<(StreamingConfig, InferenceConfig), LabelSetFactsError>`
247+
added to `pipeline.rs` alongside `run_all_exact_pipeline()`. The workload's `metrics:` hints
248+
are required and supply the label schema; every workload metric must have one. `facts`
249+
(`LabelSetFacts`) supplies each AQE's group cardinalities and series count; arrival rate is
250+
derived as `series_count × 1000 / scrape_interval_ms`, assuming one sample per series per scrape.
227251

228252
`translator.rs::build_inference_config()` now populates `query_configs`: for each assignment with
229253
a real `aggregation_id`, emits one `QueryConfig` per original query string, with
@@ -319,26 +343,44 @@ standalone `asap-optimizer-cli` binary (`asap-planner-rs/src/bin/optimizer_cli.r
319343
```
320344
cargo run -p asap_planner --bin asap-optimizer-cli -- \
321345
--input_config <path/to/workload.yaml> \
322-
--dataset <path/to/series-inventory.csv> \
346+
--label-set-facts <path/to/label_set_facts.yaml> \
323347
--data-ingestion-interval-ms 60000 \
324-
[--rho 1.0] \
325348
[--atomic-costs <path/to/atomic_costs.json> \
326349
--atomic-cost-workload <path/to/workload-selector.json>]
327350
```
328351

329-
Takes the same `ControllerConfig` YAML format as `asap-planner --input_config`.
330-
The required CSV dataset is one row per unique metric series, with a fixed `metric`
331-
column and label columns. It derives the metric schema and each AQE's distinct
332-
group count; no live Prometheus connection is needed. A `metrics:` hints block, when
333-
present, is checked against the dataset and mismatches fail loudly.
334-
Prints deployed streaming configs and query configs to stdout. `--rho` is the
335-
placeholder arrival rate (see TODOs below — not real yet). `--atomic-costs` is
352+
Takes the same `ControllerConfig` YAML format as `asap-planner --input_config`; its
353+
`metrics:` hints are required and must cover every workload metric, since they supply the
354+
label schema. `--label-set-facts` is a YAML file of externally provided facts — the optimizer
355+
never estimates them (example: `optimizer-label-set-facts.example.yaml`, for the workload
356+
`optimizer-workload.example.yaml`; its top-k query needs `--atomic-costs` with a CMS-with-heap
357+
reference row, otherwise the run fails with that item listed as unservable):
358+
359+
```yaml
360+
series: # per (metric, spatial_filter)
361+
- {metric: http_requests_total, spatial_filter: 'env="prod"', series_count: 200}
362+
groups: # per (metric, spatial_filter, grouping_labels)
363+
- {metric: http_requests_total, spatial_filter: 'env="prod"', grouping_labels: [job], cardinality: 10}
364+
```
365+
366+
- `spatial_filter` and `grouping_labels` are required; `""` means unfiltered, `[]` means
367+
full aggregation (cardinality must then be 1). Filters are normalized like the workload's.
368+
- `grouping_labels` are the AQE's resolved output labels (all labels for `*_over_time` and
369+
top-k; the `by` labels after resolving `without`). A `topk by (L)` query also needs a
370+
`groups` row for `[L]`.
371+
- `1 ≤ cardinality ≤ series_count`. Missing facts are an error listing every expected key;
372+
unused facts only warn.
373+
- Arrival rate is `series_count × 1000 / data-ingestion-interval-ms`.
374+
375+
No live Prometheus connection is needed.
376+
Prints deployed streaming configs and query configs to stdout. `--atomic-costs` is
336377
optional; when supplied it requires `--atomic-cost-workload`, a JSON file
337378
containing the exact `profiles[].workload` value from that benchmark artifact.
338379
The loader validates the document schema and rejects a selector that matches
339380
zero or multiple profiles; it never mixes entries across workloads. Omit both
340381
flags and ordinary unbenchmarked candidates use the flat stub, while
341-
CMS-with-heap candidates warn and are dropped until a matching reference row is available.
382+
CMS-with-heap candidates warn and are dropped until a matching reference row is available;
383+
an item left with no candidate makes the run fail, listing every unservable item.
342384

343385
### Running with real sketch-bench costs
344386

@@ -363,12 +405,13 @@ CMS-with-heap candidates warn and are dropped until a matching reference row is
363405
# Run the actual optimizer:
364406
cargo run -p asap_planner --bin asap-optimizer-cli -- \
365407
--input_config workload.yaml --data-ingestion-interval-ms 60000 \
366-
--dataset path/to/series-inventory.csv \
408+
--label-set-facts path/to/label_set_facts.yaml \
367409
--atomic-costs path/to/atomic_costs.json \
368410
--atomic-cost-workload path/to/workload-selector.json
369411
```
370412
371-
`candidate-gen-dump`'s output labels each resolved params row `[real]` or `[stub]`; candidates
413+
`candidate-gen-dump`'s output labels each resolved params row `[real]`, `[stub]`, or
414+
`[analytical mem, stub cpu]` (trivial accumulators); candidates
372415
whose required cost row is missing are shown as `DROPPED`. CMS-with-heap uses its fixed-top-k
373416
reference model when the `{rows,cols}` row is present and is dropped otherwise.
374417
@@ -384,9 +427,9 @@ instead of `generator::generate_plan()`, likely behind an opt-in flag first.
384427
|---|---|---|
385428
| `aqe_extractor.rs` | `extract_requirements()` | Duplicates `build_query_requirements_promql` in `asap-query-engine/src/engines/simple_engine/promql.rs:614` — extract to shared free fn in `asap_types::query_requirements` |
386429
| `capability_matching.rs` | `labels_compatible()`, line 86 | Relax to superset matching — do in Phase 3b |
387-
| `cost_model.rs` | `ingest_cost()`, `query_cost()` | Prometheus-backed `N_g` profiling remains for #525; dataset-backed `N_g` is implemented in #693 |
430+
| `cost_model.rs` | `key_tracker_ingest_cost()` | Paired DeltaSet uses analytical key bytes and stub insert CPU until sketch-bench measures it |
388431
| `cost_model.rs` | `CostWeights::default()` | Self-consistent stub ratio (mem:cpu ≈ 1e-9:1), not real $/byte-sec vs $/cpu-sec calibration |
389-
| `greedy.rs`, `pipeline.rs` | `rho_g` parameter | Single placeholder value applied uniformly; real per-config rates need Prometheus scrape-rate × active-series-count, not wired up |
432+
| `label_set_facts.rs` | `resolve()` | Arrival rate assumes one sample per series per scrape; overestimates sparse or irregular series |
390433
| `translator.rs` | `retention_count_for_assignment(Subtract)` | Returns hardcoded `1`; should be the actual checkpoint count needed to cover the full lookback |
391434
| `promql/generator.rs` | `generate_plan()` doc comment | Flags that `Controller::generate()` still uses the hardcoded path, not the optimizer — see "Offline Testing" section above |
392435
| — | Accuracy constraint | No `Error(a,g) ≤ ε_a` check exists anywhere; nothing stops picking an under-provisioned sketch (Phase 3d) |
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Example workload for asap-optimizer-cli; facts in optimizer-label-set-facts.example.yaml.
2+
# The top-k query is only servable with --atomic-costs containing a CMS-with-heap
3+
# reference row; without it the optimizer reports it as unservable.
4+
query_groups:
5+
- queries:
6+
- 'sum by (job) (http_requests_total{env="prod"})'
7+
- 'quantile_over_time(0.99, latency_ms[5m])'
8+
- 'topk by (job) (5, sum_over_time(http_requests_total{env="prod"}[5m]))'
9+
repetition_delay_ms: 60000
10+
metrics:
11+
- {metric: http_requests_total, labels: [job, instance, env]}
12+
- {metric: latency_ms, labels: [instance]}

‎Cargo.lock‎

Lines changed: 0 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎asap-common/dependencies/rs/promql_utilities/src/query_logics/logics.rs‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -86,12 +86,11 @@ pub fn does_precompute_operator_support_subpopulations(
8686
// own top-k keys via CmsHeapItem. So it supports subpopulations the
8787
// same way MultipleSum does: labels go in `aggregated`, not
8888
// `grouping` (see `set_subpopulation_labels`).
89-
AggregationType::CountMinSketchWithHeap if matches!(statistic, Statistic::Topk) => true,
89+
AggregationType::CountMinSketchWithHeap => true,
9090

91-
AggregationType::HLL => false,
92-
93-
// Default: not supported
94-
_ => panic!("Unexpected precompute operator: {}", precompute_operator),
91+
AggregationType::HLL
92+
| AggregationType::SetAggregator
93+
| AggregationType::DeltaSetAggregator => false,
9594
}
9695
}
9796

‎asap-planner-rs/Cargo.toml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,6 @@ anyhow.workspace = true
3737
tracing.workspace = true
3838
tracing-subscriber.workspace = true
3939
clap.workspace = true
40-
csv = "1.4.0"
4140
indexmap.workspace = true
4241
chrono.workspace = true
4342
promql-parser = "0.5.0"

‎asap-planner-rs/src/bin/candidate_gen_dump.rs‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,11 @@ fn main() -> anyhow::Result<()> {
8383
println!(" queries: {:?}", aqe.query_strings);
8484

8585
let candidates = enumerate_candidates(aqe, args.scrape_interval_ms);
86-
print_candidates_grouped(&candidates, &atomic_cost_table);
86+
print_candidates_grouped(
87+
&candidates,
88+
&atomic_cost_table,
89+
aqe.requirements.grouping_labels.len(),
90+
);
8791
}
8892

8993
Ok(())
@@ -96,7 +100,11 @@ fn main() -> anyhow::Result<()> {
96100
/// params (M) [× N windows = NM total]:
97101
/// ...
98102
/// EXACT is printed last as a single line.
99-
fn print_candidates_grouped(candidates: &[CandidateConfig], atomic_cost_table: &AtomicCostTable) {
103+
fn print_candidates_grouped(
104+
candidates: &[CandidateConfig],
105+
atomic_cost_table: &AtomicCostTable,
106+
n_grouping_labels: usize,
107+
) {
100108
// Collect unique (agg_type_str, sub_type) keys in first-seen order.
101109
let mut group_order: Vec<(String, String)> = Vec::new();
102110
// (agg_type_str, sub_type) -> (agg_type, unique windows, unique params)
@@ -200,10 +208,18 @@ fn print_candidates_grouped(candidates: &[CandidateConfig], atomic_cost_table: &
200208
} else {
201209
let param_map: HashMap<String, Value> =
202210
p.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
203-
match resolve_atomic_costs(atomic_cost_table, *agg_type, &param_map) {
211+
match resolve_atomic_costs(
212+
atomic_cost_table,
213+
*agg_type,
214+
&param_map,
215+
n_grouping_labels,
216+
) {
204217
Some(costs) => {
205-
let label = if costs == AtomicCosts::default() {
218+
let stub = AtomicCosts::default();
219+
let label = if costs == stub {
206220
"stub"
221+
} else if costs.insert_cpu_secs == stub.insert_cpu_secs {
222+
"analytical mem, stub cpu"
207223
} else {
208224
"real"
209225
};

0 commit comments

Comments
 (0)