Skip to content

Commit e2df4a5

Browse files
committed
docs: explain installed QueryPlan DAG execution
1 parent 215fc79 commit e2df4a5

2 files changed

Lines changed: 180 additions & 0 deletions

File tree

‎docs/design_docs/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ notes and migration gates distinguish implemented behavior from proposed changes
1212
- [Summary definitions table and SDS](summary-catalog-sds-architecture.md) owns definition
1313
and instance identity, version-scoped state references, readiness and state
1414
lifecycle semantics.
15+
- [QueryPlan DAG execution](query-dag-execution.md) explains how the query
16+
engine evaluates installed query sub-DAGs and reads bound stored summaries.
1517
- [Architecture migration delivery plan](asapplanner-migration-plan.md) defines
1618
common-library extraction, removal of ASAPCollector dependencies, the two-plan
1719
rollout, and backend acceptance/retirement gates.
Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
# QueryPlan DAG execution
2+
3+
Audience: backend designers and developers.
4+
5+
This document describes how the backend executes an installed `QueryPlan`.
6+
The [plan split](asapplanner-integration.md) defines why query-time work is
7+
separate from maintenance, and the
8+
[SDS contract](summary-catalog-sds-architecture.md) defines the stored records
9+
read by the plan.
10+
11+
## Runtime contract
12+
13+
The physical compiler publishes one `QueryPlanEntry` for each installed query.
14+
An entry contains a root, typed nodes, dependency IDs, evaluation-time rules,
15+
and an explicit fallback policy. The query engine executes the nodes reachable
16+
from that root. It does not parse the incoming expression into another runtime
17+
program and does not execute `selected_dags`; those documents retain Planner
18+
semantic provenance for validation and debugging.
19+
20+
The request first resolves a canonical query identity in the active, immutable
21+
physical-plan snapshot. The Summary Catalog, PrecomputePlan and QueryPlan in
22+
that snapshot have the same plan ID and version. A lookup miss is a capability
23+
miss and follows the installed routing policy.
24+
25+
```mermaid
26+
flowchart LR
27+
Request[Canonical query request] --> Lookup[Lookup QueryPlanEntry]
28+
Lookup --> Root[Start at entry.root]
29+
Root --> Walk[Find reachable sub-DAG]
30+
Walk --> Inputs[Evaluate dependencies]
31+
Inputs --> Node[Execute node adapter]
32+
Node --> Memo[Memoize node output]
33+
Memo --> Result[Adapt root output]
34+
Store[SummaryStore] -->|StoredOutputReference| Inputs
35+
Exact[External exact engine] -->|declared exact leaf| Inputs
36+
```
37+
38+
Installation rejects missing inputs, cycles, unreachable nodes, invalid output
39+
bindings, unsupported provenance versions, and a reader whose window contract
40+
or `StoredOutputReference` differs from its PrecomputePlan writer. Runtime
41+
errors retain the query ID and node ID.
42+
43+
## Execution layers
44+
45+
V1 uses one plan and three value adapters rather than three semantic programs.
46+
47+
| Adapter | Nodes and values | Scheduling |
48+
| --- | --- | --- |
49+
| Stored-summary adapter | `ReadMaterialization`, state merge, exact/sketch readout, scalar arithmetic and reduction | Reachable nodes run in topological order. Each node runs once for the requested root. |
50+
| PromQL/MetricsQL residual adapter | Logical aggregation, binary, temporal, subquery, candidate reranking and prepared exact leaves | Demand evaluation memoized by `(node_id, evaluation_time)`. The time key is required because a subquery evaluates one dependency at several timestamps. |
51+
| ClickHouse relation adapter | External relations, filters, projections and joins around stored-summary sub-DAGs | Demand evaluation memoized by `node_id`. Each edge validates its declared relation schema. Stored-summary sub-DAGs delegate to the topological adapter. |
52+
53+
All adapters start from a `QueryPlanEntry` node. The language adapters only
54+
represent different runtime value types. They cannot select a replacement
55+
definition or reconstruct an operator from the request text.
56+
57+
## StoredSummary reads
58+
59+
A `ReadMaterialization` carries the exact `StoredOutputReference` published for
60+
its writer. V1 derives its output ID from the definition ID because the current
61+
store index is definition keyed. The read proceeds as follows:
62+
63+
1. Validate the output reference and its definition identity.
64+
2. Use the active catalog generation and definition to select only visible
65+
stored summaries. A previous generation is visible only after an explicit
66+
compatible-definition decision during activation.
67+
3. Reject unpublished input, a request outside the bound pane/full-window
68+
phase, missing required panes, or an unavailable population.
69+
4. Check the installed catalog descriptor against the requested readout family.
70+
5. Decode payloads using their stored descriptor and merge only compatible
71+
states according to the plan's grouping contract.
72+
73+
The query engine performs an ID lookup in the SummaryStore index. It does not
74+
search the catalog at serving time for an alternative summary. Instance
75+
metadata and payload are one logical `StoredSummary`; the physical
76+
`SummaryStateReference` used by the storage engine is not a QueryPlan edge.
77+
78+
Readiness is conservative. Until the store represents completed empty panes,
79+
a missing additive pane is not assumed to be zero. A coverage or format miss
80+
therefore triggers the entry's installed fallback behavior instead of returning
81+
a partial accelerated answer.
82+
83+
## Dependency ordering and reuse
84+
85+
Each evaluation derives only the sub-DAG reachable from the requested root.
86+
Dependencies complete before their consumer. Output memoization makes a diamond
87+
graph execute its shared node once within that evaluation.
88+
89+
PromQL subqueries are the exception to a plain node-only memo key. The same
90+
node has a different result at each evaluation timestamp, so their memo key is
91+
`(node_id, evaluation_time)`. Repeated access at the same timestamp reuses the
92+
value. Prepared external leaves use the same identity and are issued before
93+
local residual evaluation so network I/O does not hide inside a synchronous
94+
operator.
95+
96+
Memoization is request local. It is discarded after the root result is adapted;
97+
the SummaryStore is the cross-request reuse boundary. A summary revision fence
98+
prevents a response from combining payloads changed during one evaluation.
99+
100+
## Exact work and fallback
101+
102+
An `ExternalExact` node is an ordinary typed DAG leaf. Its expression, output
103+
shape, parameters and input contracts are installed with the plan. The engine
104+
may prepare it through Prometheus, MetricsQL or ClickHouse only when forwarding
105+
is enabled. Candidate-dependent leaves first evaluate their installed candidate
106+
sub-DAG and pass the resulting membership set to the exact request.
107+
108+
Fallback is not an implicit parser retry inside the executor. An
109+
`ExactFallback` node fails deliberately, and the entry's `FallbackPolicy`
110+
determines whether the router may call the exact backend. Invalid graphs,
111+
incompatible stored state and unsupported nodes fail closed with a scoped
112+
reason.
113+
114+
## Shared KLL example
115+
116+
Two installed query entries can read one precomputed output while keeping
117+
different query roots:
118+
119+
```mermaid
120+
flowchart LR
121+
subgraph P[PrecomputePlan]
122+
Samples[Latency samples] --> KLL[Build KLL by service]
123+
KLL --> Write[Write output latency-kll]
124+
end
125+
Write --> Store[(StoredSummary records)]
126+
subgraph Q50[QueryPlanEntry p50]
127+
Read50[Read latency-kll] --> P50[Quantile 0.50]
128+
end
129+
subgraph Q99[QueryPlanEntry p99]
130+
Read99[Read latency-kll] --> P99[Quantile 0.99]
131+
end
132+
Store --> Read50
133+
Store --> Read99
134+
```
135+
136+
`Read50`, `Read99` and `Write` carry the same `StoredOutputReference`. Each
137+
request reads the ready population/window records and executes only its own
138+
readout sub-DAG. No query rebuilds the KLL and no serving-time catalog search
139+
chooses a different summary.
140+
141+
Within one entry, two parents may also share a read or relational node. The
142+
request-local memo returns its existing value to the second parent. Across the
143+
p50 and p99 requests, payload reuse comes from SummaryStore rather than a
144+
cross-request executor cache.
145+
146+
## Concurrency, cancellation and limits
147+
148+
Requests execute concurrently and own separate memo maps and intermediate
149+
values. The active physical plan and committed stored summaries are shared
150+
through immutable snapshots or synchronized store indexes. No mutable execution
151+
context is shared between requests.
152+
153+
V1 evaluates ready nodes sequentially inside one request. Independent requests
154+
still run concurrently. Parallel execution of independent nodes is unnecessary
155+
for correctness and remains future work. External I/O uses the request client's
156+
timeout; cancellation drops the request-local evaluation and its prepared
157+
values. Logical subqueries enforce depth and evaluation budgets to bound memory
158+
and work.
159+
160+
Large relation intermediates and a unified resource budget across all three
161+
adapters remain follow-up work. The current implementation also does not cache
162+
root results across requests, add a distributed query scheduler, or reuse stored
163+
payloads across plan versions without the activation-time compatibility check.
164+
165+
## End-to-end sequence
166+
167+
1. The control plane installs and activates one coherent physical plan.
168+
2. The serving endpoint canonicalizes the request only to find its installed
169+
entry; it does not compile a new execution graph.
170+
3. The engine validates the entry against the active catalog generation.
171+
4. Declared external leaves are prepared when allowed.
172+
5. The appropriate value adapter evaluates the reachable sub-DAG with
173+
request-local memoization.
174+
6. `ReadMaterialization` nodes resolve their bound ready stored summaries.
175+
7. Node failures carry query/node context and follow the installed fallback
176+
policy.
177+
8. A revision fence confirms that stored input did not change during execution.
178+
9. The root value is adapted to the Prometheus/MetricsQL or ClickHouse response.

0 commit comments

Comments
 (0)