@@ -35,7 +35,12 @@ The optimizer output type is `OptimizerSolution`, which is then translated into
3535| ---| ---|
3636| ` mergeable ` | Two instances can be combined → supports Merge query method |
3737| ` subtractable ` | Element-wise difference defined → supports Subtract (tumbling only) |
38- | ` subpopulation_aware ` | One instance handles multiple label-group keys → N(s,g)=1 |
38+ | ` memory_grows_with_keys ` | One instance is a keyed map whose memory grows per key (` Multiple* ` ), vs a fixed-size instance |
39+
40+ How many instances a config has comes from its ` grouping_labels ` , split the same way as the
41+ legacy planner (` planner/labels.rs::set_subpopulation_labels ` ): keyed types keep the output
42+ labels in ` aggregated_labels ` (one instance), per-group types in ` grouping_labels ` (one instance
43+ per group), and ` topk by (L) ` gets one heap per value of ` L ` .
3944
4045### Query method (not a free variable — derived from ingest_type × W vs range_a × algebra)
4146
@@ -76,7 +81,7 @@ asap-planner-rs/src/optimizer/
7681├── translator.rs translate(&OptimizerSolution) → (StreamingConfig, InferenceConfig)
7782├── aqe_extractor.rs Rqe, extract_aqes()
7883├── pipeline.rs run_all_exact_pipeline(), run_greedy_pipeline()
79- ├── dataset .rs CSV series inventory, schema validation, and N_g profiling [#693 ]
84+ ├── label_set_facts .rs external series counts / group cardinalities → per-AQE facts [#756 ]
8085├── sketch_properties.rs algebraic properties per AggregationType [Phase 2a, done]
8186├── candidate_gen.rs enumerate candidate configs per AQE [Phase 2b, done]
8287├── cost_model.rs ingest/query cost formulas [Phase 2c, done]
@@ -126,20 +131,23 @@ Each AQE gets its own best config independently (no cross-AQE sharing). MIP shar
126131#### 2a — ` sketch_properties.rs ` (done)
127132
128133``` rust
129- pub struct SketchProperties { pub mergeable : bool , pub subtractable : bool , pub subpopulation_aware : bool }
134+ pub struct SketchProperties { pub mergeable : bool , pub subtractable : bool , pub memory_grows_with_keys : bool }
130135pub fn sketch_properties (t : AggregationType ) -> SketchProperties
131136```
132137
133138Implemented values — `CountMinSketchWithHeap ` ended up `mergeable = false ` (not `true `/ unclear as
134139originally guessed ): the heap top - k list doesn 't compose across merged / subtracted windows even
135- though the underlying CMS cells would . Everything else matches the original guess :
136- - `CountMinSketch `: mergeable = T , subtractable = T , subpopulation_aware = T
137- - `CountMinSketchWithHeap `: mergeable = F , subtractable = F , subpopulation_aware = T
138- - `DatasketchesKLL `, `HydraKLL `: mergeable = T , subtractable = F , subpopulation_aware = F (true for HydraKLL too )
139- - `Sum `/ `MultipleSum `: mergeable = T , subtractable = T , subpopulation_aware = F / T respectively
140- - `HLL `, `SetAggregator `, `DeltaSetAggregator `: mergeable = T , subtractable = F , subpopulation_aware = F
141- - `MinMax `/ `MultipleMinMax `, `Increase `/ `MultipleIncrease `: mergeable = T , subtractable = F , subpopulation_aware = F / T
142- - `SingleSubpopulation `/ `MultipleSubpopulation ` (legacy wrapper types ): all false (unknown , treated conservatively )
140+ though the underlying CMS cells would . `memory_grows_with_keys ` is true only for the `Multiple * `
141+ maps :
142+ - `CountMinSketch `: mergeable = T , subtractable = T
143+ - `CountMinSketchWithHeap `: mergeable = F , subtractable = F
144+ - `DatasketchesKLL `, `HydraKLL `: mergeable = T , subtractable = F
145+ - `Sum `/ `MultipleSum `: mergeable = T , subtractable = T
146+ - `HLL `, `SetAggregator `, `DeltaSetAggregator `: mergeable = T , subtractable = F
147+ - `MinMax `/ `MultipleMinMax `, `Increase `/ `MultipleIncrease `: mergeable = T , subtractable = F
148+
149+ The optimizer never proposes single - group `Sum `/ `MinMax `/ `Increase `: with analytical per - key
150+ memory they cost the same as their `Multiple * ` forms (`OPTIMIZER_SKIPPED_AGG_TYPES `).
143151
144152When a type is both `mergeable ` and `subtractable ` (e . g. `Sum `, `CountMinSketch `),
145153`candidate_gen . rs` prefers `Subtract ` — it 's O (1 ) regardless of `n `, strictly cheaper than `Merge `'s O (n ).
@@ -151,11 +159,18 @@ pub struct CandidateConfig {
151159 pub config : Option <AggregationConfig >, // None = EXACT fallback
152160 pub query_method : QueryMethod ,
153161 pub n_windows : u64 ,
154- pub label_group_count : u64 , // dataset-derived N_g
162+ pub instance_count : u64 , // from the config's grouping labels
163+ pub output_group_count : u64 , // cardinality of the AQE's output labels
164+ pub key_config : Option <AggregationConfig >, // paired DeltaSet for CMS/HydraKLL
155165}
156- pub fn enumerate_candidates (aqe : & Aqe , scrape_interval_secs : u64 ) -> Vec <CandidateConfig >
166+ pub fn enumerate_candidates_with_facts (aqe : & Aqe , scrape_interval_ms : u64 , facts : & ItemFacts ) -> Vec <CandidateConfig >
157167```
158168
169+ CMS and HydraKLL can 't list their own keys , so each such candidate carries the same paired
170+ DeltaSetAggregator the legacy planner deploys (`needs_key_aggregation `): same labels , Tumbling
171+ at the value 's slide , retaining `⌈range_a / slide ⌉` panes . Greedy registers both and the
172+ translator references both in the query config .
173+
159174Enumeration : `compatible_agg_types (stat )` × param grid (hardcoded small grids per sketch type ,
160175see `CMS_DEPTHS `/ `CMS_WIDTHS `/ etc . constants — replace with sketch - bench sweep results in Phase 3 )
161176× window candidates × ingest type :
@@ -179,15 +194,23 @@ Label granularity: only proposes configs at the label granularity the AQE itself
179194pub struct AtomicCosts { mem_bytes_per_instance , insert_cpu_secs , merge_cpu_secs , subtract_cpu_secs ,
180195 query_cpu_secs , exact_query_cpu_secs : f64 }
181196pub struct CostWeights { ingest_mem , ingest_cpu , query_mem , query_cpu : f64 }
182- pub fn ingest_cost (candidate : & CandidateConfig , rho_g : f64 , costs : & AtomicCosts , weights : & CostWeights ) -> f64
197+ pub fn ingest_cost (candidate : & CandidateConfig , arrival_rate_hz : f64 , costs : & AtomicCosts , weights : & CostWeights ) -> f64
183198pub fn query_cost (a : & Aqe , candidate : & CandidateConfig , costs : & AtomicCosts , weights : & CostWeights ) -> f64
184- pub fn total_cost_rate (a : & Aqe , candidate : & CandidateConfig , rho_g : f64 , costs : & AtomicCosts , weights : & CostWeights ) -> f64
199+ pub fn total_cost_rate (a : & Aqe , candidate : & CandidateConfig , arrival_rate_hz : f64 , costs : & AtomicCosts , weights : & CostWeights ) -> f64
185200```
186201
187- Implements the design doc 's formulas with `N (s ,g ) = 1 ` for subpopulation - aware sketches and
188- the dataset - derived `N_g ` for per - label - group sketches . The CSV series inventory and exact
189- filter profiling are implemented in `dataset . rs` (#693 ); Prometheus - backed profiling remains
190- the follow - up in #525 .
202+ Group cardinality enters per candidate :
203+
204+ | | Multiplier |
205+ | --- | --- |
206+ | Memory , merge / subtract CPU | `output_group_count ` for keyed maps (`Multiple * `), else `instance_count ` |
207+ | Query read CPU | `output_group_count ` (one read per output group ); top - k : `instance_count ` (one heap read each ) |
208+ | Insert CPU | arrival rate `λ`, independent of cardinality |
209+ | Paired key tracker | `output_group_count ` key entries + one insert per item |
210+
211+ Trivial accumulators (`Sum `/ `MinMax `/ `Increase ` and `Multiple * `) use an analytical per- group
212+ entry, `(n_labels × 4 + value_bytes) × 8/ 7`: label values dictionary- encoded as `u32 ` codes
213+ (dictionary amortized, not charged), hashbrown load- factor slack. CPU stays on the stub.
191214`AtomicCosts ` added `exact_query_cpu_secs` (not in the original plan) — without a non- zero cost
192215for the EXACT fallback's query, `IngestCost = 0, QueryCost = 0` would make EXACT always win trivially.
193216
@@ -212,18 +235,19 @@ it then, reusing `window_compatible`, `spatial_filter_compatible`, `topk_weighti
212235#### 2e — `greedy. rs` + `pipeline. rs` + `translator. rs` (done)
213236
214237```rust
215- pub fn greedy_assign (aqes : Vec <Aqe >, scrape_interval_secs : u64 , rho_g : f64 ,
216- costs : & AtomicCosts , weights : & CostWeights ) -> OptimizerSolution
238+ pub fn greedy_assign(aqes: Vec <Aqe >, scrape_interval_ms : u64 , atomic_cost_table : & AtomicCostTable ,
239+ weights : & CostWeights , facts : & [ ItemFacts ] ) -> OptimizerSolution
217240```
218241
219242For each AQE independently: `argmin` over `enumerate_candidates(aqe, ... )` by `total_cost_rate`.
220243No feasibility filter needed (see 2d note) — every candidate from `enumerate_candidates` is valid
221244for that AQE by construction. Assigns sequential `aggregation_id`s to deployed configs.
222245
223- `run_greedy_pipeline (config , dataset , scrape_interval_secs , rho_g ) -> Result <(StreamingConfig , InferenceConfig ), DatasetError >`
224- added to `pipeline . rs` alongside `run_all_exact_pipeline ()`. The dataset is required for the
225- greedy path and supplies metric schemas plus the `N_g ` profile for each AQE . `rho_g ` is
226- currently a single placeholder value applied uniformly to every candidate — see open TODO below .
246+ `run_greedy_pipeline(config, facts, scrape_interval_ms, atomic_cost_table) -> Result <(StreamingConfig , InferenceConfig ), LabelSetFactsError >`
247+ added to `pipeline. rs` alongside `run_all_exact_pipeline()`. The workload's `metrics: ` hints
248+ are required and supply the label schema; every workload metric must have one . `facts `
249+ (`LabelSetFacts `) supplies each AQE 's group cardinalities and series count ; arrival rate is
250+ derived as `series_count × 1000 / scrape_interval_ms `, assuming one sample per series per scrape .
227251
228252`translator . rs :: build_inference_config ()` now populates `query_configs `: for each assignment with
229253a real `aggregation_id `, emits one `QueryConfig ` per original query string , with
@@ -319,26 +343,44 @@ standalone `asap-optimizer-cli` binary (`asap-planner-rs/src/bin/optimizer_cli.r
319343```
320344cargo run - p asap_planner -- bin asap- optimizer- cli -- \
321345 -- input_config <path/ to/ workload. yaml> \
322- -- dataset <path/ to/ series - inventory . csv > \
346+ -- label - set - facts <path/ to/ label_set_facts . yaml > \
323347 -- data- ingestion- interval- ms 60000 \
324- [-- rho 1. 0] \
325348 [-- atomic- costs <path/ to/ atomic_costs. json> \
326349 -- atomic- cost- workload <path/ to/ workload- selector. json>]
327350```
328351
329- Takes the same `ControllerConfig ` YAML format as `asap- planner -- input_config`.
330- The required CSV dataset is one row per unique metric series, with a fixed `metric`
331- column and label columns. It derives the metric schema and each AQE 's distinct
332- group count; no live Prometheus connection is needed . A `metrics : ` hints block , when
333- present , is checked against the dataset and mismatches fail loudly .
334- Prints deployed streaming configs and query configs to stdout . `-- rho ` is the
335- placeholder arrival rate (see TODOs below — not real yet ). `-- atomic - costs ` is
352+ Takes the same `ControllerConfig ` YAML format as `asap- planner -- input_config`; its
353+ `metrics : ` hints are required and must cover every workload metric , since they supply the
354+ label schema . `-- label - set - facts ` is a YAML file of externally provided facts — the optimizer
355+ never estimates them (example : `optimizer - label - set - facts . example. yaml`, for the workload
356+ `optimizer - workload . example. yaml`; its top - k query needs `-- atomic - costs ` with a CMS - with - heap
357+ reference row , otherwise the run fails with that item listed as unservable ):
358+
359+ ```yaml
360+ series : # per (metric , spatial_filter )
361+ - {metric : http_requests_total , spatial_filter : 'env = " prod" ', series_count: 200}
362+ groups: # per (metric, spatial_filter, grouping_labels)
363+ - {metric: http_requests_total, spatial_filter: ' env = " prod" ', grouping_labels: [job], cardinality: 10}
364+ ```
365+
366+ - `spatial_filter` and `grouping_labels` are required; `""` means unfiltered, `[]` means
367+ full aggregation (cardinality must then be 1). Filters are normalized like the workload' s .
368+ - `grouping_labels ` are the AQE 's resolved output labels (all labels for `* _over_time ` and
369+ top - k ; the `by ` labels after resolving `without `). A `topk by (L )` query also needs a
370+ `groups ` row for `[L ]`.
371+ - `1 ≤ cardinality ≤ series_count `. Missing facts are an error listing every expected key ;
372+ unused facts only warn .
373+ - Arrival rate is `series_count × 1000 / data - ingestion - interval - ms `.
374+
375+ No live Prometheus connection is needed .
376+ Prints deployed streaming configs and query configs to stdout . `-- atomic - costs ` is
336377optional ; when supplied it requires `-- atomic - cost - workload `, a JSON file
337378containing the exact `profiles []. workload` value from that benchmark artifact .
338379The loader validates the document schema and rejects a selector that matches
339380zero or multiple profiles ; it never mixes entries across workloads . Omit both
340381flags and ordinary unbenchmarked candidates use the flat stub, while
341- CMS - with- heap candidates warn and are dropped until a matching reference row is available.
382+ CMS - with- heap candidates warn and are dropped until a matching reference row is available;
383+ an item left with no candidate makes the run fail , listing every unservable item .
342384
343385### Running with real sketch - bench costs
344386
@@ -363,12 +405,13 @@ CMS-with-heap candidates warn and are dropped until a matching reference row is
363405# Run the actual optimizer:
364406cargo run -p asap_planner --bin asap-optimizer-cli -- \
365407 --input_config workload.yaml --data-ingestion-interval-ms 60000 \
366- -- dataset path/ to/ series - inventory . csv \
408+ --label-set-facts path/to/label_set_facts.yaml \
367409 --atomic-costs path/to/atomic_costs.json \
368410 --atomic-cost-workload path/to/workload-selector.json
369411```
370412
371- `candidate- gen - dump`'s output labels each resolved params row `[real]` or `[stub]`; candidates
413+ `candidate-gen-dump`'s output labels each resolved params row `[real]`, `[stub]`, or
414+ `[analytical mem, stub cpu]` (trivial accumulators); candidates
372415whose required cost row is missing are shown as `DROPPED`. CMS-with-heap uses its fixed-top-k
373416reference model when the `{rows,cols}` row is present and is dropped otherwise.
374417
@@ -384,9 +427,9 @@ instead of `generator::generate_plan()`, likely behind an opt-in flag first.
384427|---|---|---|
385428| `aqe_extractor.rs` | `extract_requirements()` | Duplicates `build_query_requirements_promql` in `asap-query-engine/src/engines/simple_engine/promql.rs:614` — extract to shared free fn in `asap_types::query_requirements` |
386429| `capability_matching.rs` | `labels_compatible()`, line 86 | Relax to superset matching — do in Phase 3b |
387- | `cost_model . rs` | `ingest_cost ()`, ` query_cost ()` | Prometheus - backed ` N_g ` profiling remains for # 525 ; dataset - backed ` N_g ` is implemented in # 693 |
430+ | `cost_model.rs` | `key_tracker_ingest_cost ()` | Paired DeltaSet uses analytical key bytes and stub insert CPU until sketch-bench measures it |
388431| `cost_model.rs` | `CostWeights::default()` | Self-consistent stub ratio (mem:cpu ≈ 1e-9:1), not real $/byte-sec vs $/cpu-sec calibration |
389- | `greedy . rs`, ` pipeline . rs` | `rho_g ` parameter | Single placeholder value applied uniformly ; real per - config rates need Prometheus scrape - rate × active - series - count , not wired up |
432+ | `label_set_facts .rs` | `resolve()` | Arrival rate assumes one sample per series per scrape; overestimates sparse or irregular series |
390433| `translator.rs` | `retention_count_for_assignment(Subtract)` | Returns hardcoded `1`; should be the actual checkpoint count needed to cover the full lookback |
391434| `promql/generator.rs` | `generate_plan()` doc comment | Flags that `Controller::generate()` still uses the hardcoded path, not the optimizer — see "Offline Testing" section above |
392435| — | Accuracy constraint | No `Error(a,g) ≤ ε_a` check exists anywhere; nothing stops picking an under-provisioned sketch (Phase 3d) |
0 commit comments