Repository navigation
Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672
Description
Problem
A ranked top-k query loses the columns that say which items are ranked.
- SQL.
SELECT port, COUNT(*) AS c FROM t GROUP BY port ORDER BY c DESC LIMIT 5is rewritten by IR canonicalization (try_promote_additive_top_rankingincrates/types/src/ir/canonicalize.rs, since feat(ir): add canonicalization, sub-DAG sharing and flat DAGs on the unified IR #645) into a globalAggregate{[TopK]}. Its declared output schema is[topk_5], so the result has five counts and noportcolumn. The query asked for[port, c]. - PromQL.
topk(5, sum by (job) (m))declares[topk_5]in the same way, so thejoblabels are missing. - Every candidate inherits it. feat(planner): physical ASAP-aware optimization and plan selection for candidate logical ASAP DAGs #576 makes every candidate's root follow the frontend's declared schema
[keys.., topk_k], so the exact plan and the sketch plans all inherit the missing item columns.
Related problem in candidate composition (#575)
The top-k sketch update in composition uses the sample value as its weight, and the composed heavy-hitter candidate (e.g. CmsWithHeap on the top-k target) resolves its item and value columns by name:
- with an Int64 key or several keys (
GROUP BY host, port), composition fails with "summary input column does not resolve"; - with a key column named
value, the summary would read the key as the value.
Expected
The top-k output schema is [partition keys.., item columns.., ranked value], with the original names (SQL: port, c; PromQL: the labels plus value).
- Canonicalization keeps the item columns.
- Composition's sketch update takes its item from the item columns and its weight from the ranked measure (the count or sum), not by name.
- Physical ASAP-aware optimization (exact Sort + Limit) and every summary candidate return that same schema.
Done when
- The SQL and PromQL queries above return the item columns in every candidate, and the results match DataFusion and Prometheus semantics.
- Heavy-hitter sketch candidates compose for Int64 keys, multi-column keys, and a key named
value.
Found in the #576 review. Related: #575, #576, #645, #669 (top-k membership guarantee).
Activity
zzylol commentedon Oct 13, 2026
Related, found in the #576 review: when the top-k promotion in
canonicalize.rsrewrites a sorted query, it discards the sort's NULL ordering. DataFusion'sORDER BY s DESCis NULLS FIRST, but physical ASAP-aware optimization rebuilds the exact sort withnulls_first: false.Example: in
SELECT host, SUM(lat) AS s FROM t GROUP BY host ORDER BY s DESC LIMIT 5with a nullablelat, a group whose sum is NULL comes first in the real query, but last (or not at all) in the exact candidate.Fix in canonicalization: either refuse to promote when the ranked measure can be NULL, or carry the null ordering into the top-k so every candidate keeps it.
feat(planner): run the planning stages as the planner pipeline; retir…
test: note that the does-not-compose tests rely on #672
test: note that the does-not-compose tests rely on #672
test: note that the does-not-compose tests rely on #672
fix(planner): price state by its size; size KLL for δ (#682)