feat(planner): Stage 1 Pass 1 per-computation candidate generation

[zzylol]

Why

#509 Stage 1 starts with Pass 1, per-computation candidate generation. docs/design_docs/proposals/planner-layering.md says:

For each eligible sub-DAG, Pass 1 identifies its computation semantics, applies rewrite rules, and generates every candidate that is not provably unable to meet its accuracy requirement.

A sub-DAG is eligible when its root is an aggregation (an Aggregate operator) whose intent has at least one realization in the summary-family capabilities F: exact accumulator state or a summary family. Nested aggregations are separate targets: in sum by (job) (rate(x[1m])), the per-series rate and the per-job sum are each a target, and their options multiply.

• Pruning removes an invalid candidate. Any stage may prune, but only when the candidate is provably invalid (for example, a summary family that cannot meet the query's accuracy target), and every pruned candidate carries a reason.
• Selection discards valid candidates. Only stage 3 does this, when it picks the cheapest one.

The planner at the time (the legacy search, MajorPass, #542; retired in #674) built replacements and ranked them by cost in one search. There is no logical-only step that lists every valid per-computation candidate over the unified IR without a cost model.

What

A new module, asap_aware_mapping::per_computation_candidates, that implements Stage 1 (logical ASAP-aware optimization), Pass 1: per-computation candidate generation from planner-layering.md, for a workload of QueryRoots. It takes no cost model, accuracy model, capabilities, storage policy or materialization assignment, and does not rank, size or build replacement sub-DAGs.

• Candidates (ComputationCandidate):
  • Exact: plain exact execution of the original sub-DAG.
  • ExactAccumulator { kind, params }: an exact mergeable accumulator state (count, sum, min, max, rate, irate, increase).
  • Summary(SketchAlgorithm): a summary family, unsized. Pass 2 may share a summary and tighten it to the strictest consumer, so sizing happens later.
  • AvgAsSumCount: avg rewritten as exact sum and count accumulator states.
• One exact option per target, listed first: the exact accumulator form (ExactAccumulator, or AvgAsSumCount for avg) when the intent has one and it can be built, otherwise Exact. Offering both would duplicate "accumulator, not materialized"; Stage 2/3 decide materialization and sharing. Summary families follow in stable catalog order, which carries no preference.
• Pruning with reasons (PruneReason): RequiresExact (an exact request does not get approximate summaries) and NullableAvgInput (count counts rows, so avg over a possibly-null input is not rewritten; plain Exact becomes the target's exact option). ComputationChoices { candidates, pruned } keeps both lists.
• Targets (ComputationTarget { target, measure, choices, filter, having }): one per measure of every reachable Aggregate whose intent has a realization besides plain exact execution (even if all of them are pruned). Each measure of a multi-measure aggregate is its own target. Nested aggregates are separate targets. Producers read by scalar roots and scalar expressions are discovered, and a shared producer is one target (pointer identity).
• Filters and HAVING keep every candidate. The target records how they are applied; composition (feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575) applies them.
  • filter: the measure's own filter, applied to the state as SummaryAgg.filter.
  • having: one HavingEntry { filter, predicate, placement } per HAVING: the aggregate's own having (no consumer Filter), then one per consumer Filter directly above the aggregate (the form the SQL frontend lowers HAVING to). Only this adjacent Filter is recorded.
  • HavingPlacement::BelowAggregate only when the predicate reads only by(...) group keys and, for a consumer Filter, every consumer of the aggregate applies the same predicate; it is then pushed below as a row predicate. Everything else is AboveOutput: a Filter above the finalize/estimate output. A global aggregate's HAVING is always AboveOutput, since its one output row exists even with no input rows.
  • Approximate candidates under a value predicate get guarantee = None through feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575; multi-level push-down is Pass 1: push down filters and HAVING across levels to create more candidates (and reuse DataFusion's filter push-down) #664.
• Input checks (PerComputationCandidateError): Structure (QueryRoot::validate_structure failed), ContainsAsap (input already has ASAP operators), AssignedPhysicalAnnotation (a node already has an execution timing or a materialization), InvalidAccuracy { root, measure } (an approximate target needs epsilon and delta in (0, 1)).
• Docs: the module doc states the stage and pass and the rules above; there is no separate develop doc. planner-layering.md now says a filter or HAVING does not prevent the accumulator or summary candidates, and Example 1's diagram says Sum has no summary family instead of showing pruned summaries.

How

generate_per_computation_candidates(roots) validates each root, walks every operator node reachable from its operator roots (including those referenced by scalar expressions) once, rejects physical annotations and ASAP operators, and records each node's consumers. For each Aggregate it computes the HAVING entries from the consumers, then calls computation_candidates_for_intent(intent) per measure, using the existing accuracy_target, accuracy_budget and summary_candidates from replacement.rs. The roots are returned unchanged with the targets.

Public API:

pub fn computation_candidates_for_intent(intent: &AggIntent) -> Result<ComputationChoices, PerComputationCandidateError>;
pub fn generate_per_computation_candidates<Id: Debug>(
    roots: Vec<(Id, QueryRoot)>,
) -> Result<PerComputationCandidates<Id>, PerComputationCandidateError>; // { roots, targets }

Before this PR

No API lists Pass 1 candidates over OperatorNode / QueryRoot. The only candidate generation is the legacy cost-ranked search.

After this PR

Callers can get, for a lowered workload, every computation (target) with its unranked candidates, its pruned candidates and reasons, and how its filters and HAVING are applied. On main this module is Stage 1 Pass 1 of the planner pipeline (run_stage1_to_stage3 / Stage1ToStage3Pass, #674 and #675), and its output is the run's stage1_pass1_per_computation.

Examples (approximate = EpsilonDelta { 0.05, 0.01 }):

Measure	Candidates	Pruned
count, approximate	ExactAccumulator{Count}, CMS, Count Sketch, UnivMon	—
distinct count of one column, approximate (#509 Example 2)	Exact, HLL, Theta, KMV, UnivMon	—
distinct count of a column tuple, approximate	Exact, HLL, Theta, KMV (UnivMon not declared)	—
L2 / entropy, approximate	Exact, UnivMon	—
quantile, approximate	Exact, KLL, DDSketch	—
quantile, exact	Exact	KLL, DDSketch (RequiresExact)
TopK, approximate	Exact, CMS with heap, Count Sketch with heap	—
avg over a non-null input	AvgAsSumCount	—
avg over a nullable input	Exact	AvgAsSumCount (NullableAvgInput)
stddev	not a target

#509 Example 1 with exact accuracy: Q1 sum by (job) (rate(http_requests_total[1m])) gives two targets, ExactAccumulator{Sum} and ExactAccumulator{Rate}; Q2 topk by (job) (10, sum_over_time(http_requests_total[1m])) gives Exact for the top-k with both heap summaries pruned RequiresExact, and ExactAccumulator{Sum} for sum_over_time.

Evidence

Execution examples: crates/asap-aware-mapping/tests/per_computation_candidates.rs and crates/integration-tests/tests/per_computation_candidates_sql.rs (SQL HAVING src = 'a' → BelowAggregate; HAVING count(*) > 10 → AboveOutput; global HAVING false → AboveOutput). Screenshots and measurements: not applicable.

Verification

• Unit and integration tests: the unit test approximate_count_keeps_one_exact_accumulator_and_universal_choices; in tests/per_computation_candidates.rs: catalogs (cardinality_keeps_exact_specialized_and_universal_alternatives, frequency_statistics_keep_universal_choices, exact_quantile_stays_exact_and_approximate_keeps_both_families, topk_keeps_both_specialized_heap_choices, summary_candidates_are_unsized, avg_rewrites_to_sum_and_count_states), targets (scalar_root_producers_are_discovered_once, multi_measure_aggregate_has_one_target_per_measure, aggregate_without_realization_is_not_a_target), filters and HAVING (filtered_measure_keeps_candidates_and_records_its_filter, key_only_having_is_pushed_below_and_keeps_candidates, value_having_stays_above_output_and_keeps_candidates, having_on_one_consumer_of_a_shared_aggregate_stays_above, global_aggregate_having_stays_above), input checks (assigned_physical_annotation_asap_input_and_invalid_accuracy_are_rejected).
• End-to-end tests: example_1_exact_q1_keeps_exact_accumulators_only and example_1_exact_q2_prunes_topk_heaps lower docs: propose workload-wide planning, summary sharing, and materialization #509 Example 1 through the PromQL frontend; the three SQL HAVING tests lower through the SQL frontend.
• Other checks on the PR head: cargo fmt --check; workspace all-target Clippy with -D warnings; cargo test --workspace (1,568 passed, 0 failed, 2 ignored); dag-viewer test_render (29 passed); tools/verify_metricsql_external_consumer.sh OK.

Architectural decisions

• No cost model in Pass 1. Only stage 3 discards valid candidates (docs: propose workload-wide planning, summary sharing, and materialization #509), so the API has no cost or accuracy model and returns an unranked list.
• Summaries are unsized here; sizing happens after Pass 2 sharing.
• One exact option per target (accumulator form when buildable, else plain Exact).
• Pruned candidates are kept with a reason, as docs: propose workload-wide planning, summary sharing, and materialization #509 requires for pruning.
• Filters and HAVING are recorded, not applied: they never remove candidates; composition (feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575) applies them.

Limitations and follow-up

• feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575: compose each candidate into a replacement sub-DAG, apply the recorded filters and HAVING, and propagate guarantees (including guarantee = None under value predicates).
• feat(planner): Pass 1 offers HydraCms for grouped approximate counts #605: the grouping-level TopK Hydra candidate ("Hydra over all groups").
• Pass 1: push down filters and HAVING across levels to create more candidates (and reuse DataFusion's filter push-down) #664: HAVING push-down through more than one level.
• Rewrite rules beyond avg (for example recognizing entropy / L2 forms) and extra candidates for opaque Extension intents are not added.

Later changes: #575 added composition (compose_logical_candidate) next to Pass 1; #674 made Pass 1 the start of the planner pipeline, replacing the legacy search (MajorPass), and #675 named the pipeline run_stage1_to_stage3 / Stage1ToStage3Pass.

Stack: tracker #528 · main → #542 → #665 → #666 → #543 → #667 → #561.

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

[zzylol]

Parked as draft until Phase C (see #528). Stage 1 Pass 1 local alternatives belong to the #509 end-to-end work, which comes after finishing #511 (Phase A) and the #572 reorganization (Phase B). It will move into the logical-optimizer crate then.

🤖 Generated with Claude Code