Skip to content

Commit 4be85e7

Browse files
committed
docs: clarify precomputation terminology and remove stale review notes
1 parent 96251f7 commit 4be85e7

9 files changed

Lines changed: 74 additions & 205 deletions

File tree

‎docs/01-getting-started/architecture.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,5 +113,5 @@ Planner/ERP inputs through shared SDS descriptors, PrecomputePlan, physical
113113
instances and QueryPlan. Remote-write queue admission is not durable raw-data
114114
commit. Finite drain and typed producer-watermark completion are different
115115
protocols; the sample stream does not supply an authoritative watermark.
116-
See [maintenance replay](../developer_docs/maintenance-replay.md) for the
116+
See [maintenance replay](../developer_docs/precomputation-replay.md) for the
117117
current durable and continuous boundaries.

‎docs/design_docs/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ integration.
99
by the following three designs.
1010
- [Binding Planner Physical DAGs to deployment plans](asapplanner-integration.md)
1111
defines how backend source/state bindings and operational policy instantiate
12-
Planner-provided maintenance and query computation.
12+
Planner-provided precomputation and query computation.
1313
- [Summary definitions table and SDS](summary-catalog-sds-architecture.md) owns definition
1414
and instance identity, version-scoped state references, read eligibility and
1515
committed-state metadata.

‎docs/design_docs/physical-operators.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ be changed and tested together in Planner.
1414
The query and precompute integration PRs both use the independent ASAP DAG runtime. Deployment code binds input
1515
sources, storage, ingestion windows, publication and protocol outputs. Execution
1616
phase belongs to the physical node's data state, not the operator payload.
17-
Computation has the same semantics at ingestion time and query time.
17+
Computation has the same semantics during precomputation and query execution.
1818

1919
Candidate pruning uses a general semi-join with explicit matching keys, followed
2020
by grouped Sort and grouped Limit. The completeness certificate belongs to the
@@ -39,7 +39,7 @@ the backend can execute arbitrary raw-only installed plans.
3939

4040
Planner PR #462 owns the library and depends on Planner #461, including its
4141
composed candidate-pruning API. Backend #774 consumes the pinned library;
42-
#763 integrates ingestion DAG execution and #765 integrates query DAG execution.
42+
#763 integrates precomputation DAG execution and #765 integrates query DAG execution.
4343
The remaining backend stack builds on those integrations. #728 checks candidate
4444
structure, #742 checks selection with synthetic costs, and #775 checks installed
4545
plans against data-plane results. Production evidence and performance validation
@@ -61,7 +61,8 @@ The backend `DeploymentPlanCompiler` binds declared historical query delay to
6161
retention; it does not change a logical selector's lookback. Current-series
6262
populations retain bounded versions only when the deployment requests historical
6363
coverage. Their current and retained versions share the population memory budget.
64-
Late revisions invalidate old coverage; missing or evicted state fails explicitly.
64+
Changed inputs require a new output revision; an older common snapshot remains
65+
eligible while it satisfies freshness. Missing or evicted state fails explicitly.
6566
Candidate costing includes version residency, copying and retirement.
6667

6768
Every installed range evaluation uses the shared DAG execution path and reports
@@ -78,6 +79,7 @@ its labels from the candidate side. The native join performs the comparison.
7879

7980
Physical execution errors retain their original cause. Memory exhaustion and
8081
cancellation terminate both instant and range requests; routing does not try a
81-
second engine or exact fallback. Physical fragments in one query evaluation share
82-
one run context. This does not yet account for every protocol-buffer allocation
83-
or provide an HTTP-disconnect cancellation mechanism.
82+
second engine or exact fallback. All range steps and nested physical executions share one request budget and
83+
cancellation signal, while retaining separate execution state. Tracked inputs,
84+
workspace and results count against that budget; estimates are not a hard RSS
85+
limit. See [query execution contracts](query-dag-execution.md#request-consistency-and-resource-contracts).

‎docs/design_docs/query-dag-execution.md‎

Lines changed: 14 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
# Shared physical DAG execution
1+
# Query DAG execution
22

33
## Decision and ownership
44

55
ASAPPlanner owns `asap-physical-operators`, its independent DAG runtime and the
6-
post-ASAP IR that defines its computation contract. Precompute engine and query
6+
post-ASAP IR that defines its computation contract. The precomputation engine and query
77
engine consume the same library. Changes to an IR operation and its execution
88
implementation can be reviewed in one Planner PR.
99

@@ -22,14 +22,14 @@ custom summary state formats. Those states need not fit Arrow RecordBatch.
2222
A plan is a DAG of typed operations. A producer can have multiple consumers and
2323
executes once per run. Each consumer sees its output independently. The runtime
2424
owns dependency scheduling, bounded buffering, cancellation and request-local
25-
caching of intermediate results. Separate query evaluation times and ingestion
25+
caching of intermediate results. Separate query evaluation times and precomputation
2626
windows have separate execution state.
2727

28-
Every computation operation can execute at ingestion time or query time.
29-
The phase belongs to the plan, not the operator kind. Ingestion time includes
30-
background work triggered by incoming data. Sources and publication remain
31-
engine responsibilities; phase assignment alone does not provide a missing
32-
source or implementation.
28+
Ingestion means accepting input data. Precomputation means executing a selected
29+
Physical DAG before a query request, including work triggered by ingestion or
30+
scheduled window processing. Query execution evaluates the selected request-time
31+
DAG. Build, merge and readout are reusable operators; their placement belongs to
32+
the selected plan, not the operator kind.
3333

3434
```mermaid
3535
flowchart LR
@@ -107,42 +107,10 @@ Relation binding checks the same native implementations at installation and
107107
execution. Unsupported operations must be rejected explicitly. Declaring an
108108
external source is a separate deployment choice, not evidence of local support.
109109

110-
This table lists each operation once. Shared computation can be used in either
111-
phase; the backend column identifies the available deployment bindings.
112-
113-
| Physical operation | Purpose | Shared library implementation | Backend integration | Missing coverage |
114-
| --- | --- | --- | --- | --- |
115-
| Raw Scan | Read raw input rows | Engines can supply typed batch sources | Local raw Scan is rejected | Local raw source is explicitly deferred |
116-
| Read materialization | Load previously computed state | Shared decoding and delta reconstruction | Catalog/store references, population and window checks | Missing or incompatible state is a runtime error |
117-
| Maintain current-series state | Update values and timestamps of maintained time series | No general table update operation | Remote-write storage path | General row updates; this is not a raw-table executor |
118-
| Read current-series state | Read maintained current values | Typed source interface | Installed identity/capacity and time scope | Arbitrary raw-table reading |
119-
| Scalar | Produce a scalar | Typed source, including nullable values | Query scalar source | General ingestion literal binding |
120-
| Binary | Combine or compare values | Planner numeric/comparison kinds, typed expressions and checked-division domains | Query native Project; ingestion uses shared arithmetic kernels | Arbitrary vector matching and unsupported domains |
121-
| Unary negate | Negate a value | Int64/Float64 expression, null propagation and checked integer overflow | Query native Project | General ingestion expression binding |
122-
| Vector to scalar | Convert an instant vector to a scalar | Native Float64 operator; zero or multiple rows yield NaN | Query native batch binding | General ingestion value binding |
123-
| Exact aggregate | Compute Count/Sum/Avg/Min/Max, including ReduceSum | Native grouped aggregation; checked Int64 Sum, Float64 Sum/Avg, Int64 Count, ordered Min/Max and nullable inputs | Query vector and SQL relation bindings use shared computation | Other aggregate intents, unresolved grouping-without and per-entity relation lowering; no spill |
124-
| Finalize exact accumulator / ExactReadout | Obtain an exact result from state | Shared state merge/finalization and typed batch readout | Ingestion finalization and stored-query readout use shared kernels | Arbitrary state conversion and unsupported exact families |
125-
| Project | Select or compute output columns | Planner scalar expressions and native projection | SQL native DAG; vector expression binding | Unsupported expression functions/types |
126-
| Filter | Keep rows satisfying a predicate | Planner predicates, three-valued boolean logic and native filtering | SQL native DAG | Unsupported predicates/coercions |
127-
| Relational join, including semi-join | Match rows; semi-join retains matching left rows | Inner/left/right/full/cross/semi/anti joins; typed predicates and nullable outer outputs | SQL native DAG; candidate vectors use general semi-join with explicit keys | General ingestion source binding; SQL candidate-pruning source binding |
128-
| Sort | Order rows or values | Stable grouped sort with null placement | Query vector and SQL native Sort | Sort expressions must be projected first; unsupported key types; no spill |
129-
| Limit | Keep a bounded slice per group | Grouped offset/limit across batches | Planner grouped Limit; query vector and SQL native binding | Limit does not rank or match candidate keys |
130-
| Union | Combine compatible input streams | Fair polling of native input streams | Multi-input summary merge | General installed batch-source binding |
131-
| SummaryAgg | Construct summary state | Native builders for exact Sum/Count/Min/Max/Rate/Increase, KLL, DDSketch and HLL; other admitted families have shared update kernels | Ingestion DAG and per-window update paths | General query builder binding; native Int64/keyed updates and other batch families |
132-
| SummaryMerge | Combine compatible summary states | Native grouped merge and shared stored-state merge kernels | Ingestion native DAG; query selects panes and invokes shared state kernels | Cross-family conversion is not a merge; not every stored encoding has a native batch binding |
133-
| SummaryEstimate | Read an approximate result | Shared Planner SketchQuery dispatch; native KLL/DDSketch/HLL batch readout | Stored-state query readout uses shared kernels | Unsupported family/readout combinations; accuracy evidence remains required |
134-
| SummaryJoin | Combine summaries using summary join semantics | No registered implementation | Rejected | Concrete semantics and implementation |
135-
| SummarySubtract | Subtract summary state | No registered implementation | Rejected | Concrete semantics and implementation |
136-
| SummaryDelete | Remove state contributions | No registered implementation | Rejected | Concrete semantics and implementation |
137-
| Temporal computation | Compute Rate/Increase/Sum/Avg/Min/Max/Count over a window | Native window operator using Planner intents; reset-aware rates, Int64 Count | Query range vectors bind to native batches | Other temporal intents; general ingestion window binding |
138-
| Histogram quantile | Interpolate a quantile from histogram buckets | Native window operator using Planner HistogramQuantile intent | Query bucket input binds to native batches | General ingestion histogram source binding |
139-
| Subquery | Evaluate an expression over a time grid | Shared DAG execution and request-local caching of intermediate results | Backend binds a bounded grid of operation/evaluation-time nodes | General ingestion time-grid binding |
140-
| Extension | Execute an additional operation | No universal executor | Unsupported extensions are rejected | A registered implementation for each admitted extension |
141-
142-
Stored-state codecs and readout kernels are library capabilities; they do not
143-
imply that every family has a native batch builder. Parameter, schema, population
144-
and window compatibility remain mandatory. Availability of an approximate
145-
kernel does not prove its accuracy guarantee.
110+
Operator support belongs to the [Planner library contract](physical-operators.md),
111+
not a second backend operator matrix. Deployment acceptance additionally requires
112+
compatible typed inputs, stored-state formats, grouping, windows, and accuracy
113+
evidence. A library kernel alone does not establish deployment feasibility.
146114

147115
## Candidate pruning and grouped ranking
148116

@@ -189,13 +157,13 @@ through fallback routing or by tests supplied with already decoded raw batches.
189157
## Planner compilation and deployment binding
190158

191159
The backend `DeploymentPlanCompiler` establishes state identities, window contracts
192-
and query/precompute bindings. ASAPPlanner #462 owns `physical_planner`, native
160+
and query and precomputation bindings. ASAPPlanner #462 owns `physical_planner`, native
193161
operators and DAG execution. Its compiler accepts typed input contracts without
194162
live readers and produces `CompiledPhysicalDag`; instantiation binds inputs and
195163
checks their properties without repeating operator lowering.
196164

197165
The SQL relation adapter composes Planner-compiled operators into that reusable
198166
physical representation before reading storage, then resolves typed inputs and
199-
instantiates the selected graph. PromQL vector and maintenance adapters continue
167+
instantiates the selected graph. PromQL vector and precomputation adapters continue
200168
to supply protocol/window inputs to the same native library. Source coverage,
201169
revision admission, publication and serving remain deployment responsibilities.

‎docs/developer_docs/control-plane/physical-compiler.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ for every target collector. Legacy
4242
they are not a second semantic planner.
4343

4444
ASAPPlanner owns semantics and physical candidate construction, including summary
45-
families, maintenance lifecycles and physical boundaries. Backend evaluates
45+
families, precomputation update lifecycles and physical boundaries. Backend evaluates
4646
candidate feasibility and complete workload-scoped costs, then selects and binds
4747
the deployment without changing its computation.
4848
Missing or stale implementation evidence makes the candidate unavailable; the
@@ -117,7 +117,7 @@ Why this interface exists: it prevents adapters, protocols, and physical
117117
planning from each implementing their own query-to-summary mapping.
118118

119119
Snapshot and HTTP adapters provide the same `WindowCostModel`. After logical
120-
selection, one generator enumerates layouts allowed by each state's maintenance
120+
selection, one generator enumerates layouts allowed by each state's precomputation
121121
requirements and deployment target, then derives or matches layout-specific costs.
122122
See [repeated window planning](../planning/repeated-dashboard-panes.md) for inputs,
123123
migration and cadence examples.
@@ -228,8 +228,9 @@ Graph traversal is separate from node definitions and store semantics.
228228
Activation validates roots, edges, bindings, reachability, and cycles.
229229
The shared physical DAG runtime creates one producer per reachable node and
230230
shares its outputs with bounded buffering within the request,
231-
so a shared node in a diamond DAG performs one store/operator execution. A
232-
typed node failure follows the entry's explicit fallback route.
231+
so a shared node in a diamond DAG performs one store/operator execution. Only an
232+
eligible availability failure follows the entry's explicit fallback route.
233+
Resource exhaustion and cancellation terminate execution without fallback.
233234

234235
### What the compiler puts in CollectorPlan
235236

0 commit comments

Comments
 (0)