Skip to content

Commit 12896bd

Browse files
committed
Merge branch 'restack/pr737-main' into impl/sds-semantic-identity
# Conflicts: # control_plane/src/physical/backend_stage.rs # control_plane/src/physical/compiler.rs # control_plane/src/physical/erp.rs # control_plane/src/query_plan/residual.rs # data_plane/tests/e2e_controller_plans_and_backend_serves.rs # data_plane/tests/support/current_series_process.rs # data_plane/tests/support/issue_701_702_process.rs # docs/design_docs/asapplanner-integration.md # docs/design_docs/asapplanner-migration-plan.md # docs/design_docs/planner-backend-glossary.md # docs/design_docs/summary-catalog-sds-architecture.md
2 parents fd3a02a + bccb189 commit 12896bd

4 files changed

Lines changed: 156 additions & 21 deletions

File tree

‎docs/design_docs/asapplanner-integration.md‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,13 @@ Deployment at e9390031](https://github.com/ProjectASAP/ASAPPlanner/blob/e9390031
3535

3636
```text
3737
ASAPPlanner
38-
Logical Post-ASAP DAG + selected Summary Maintenance Lifecycle
38+
Logical Post-ASAP candidates + legal Summary Maintenance Lifecycles
3939
↓ Physical Plan Compiler
40-
Physical DAGs + typed input/output boundaries
40+
Supported physical candidates + typed input/output boundaries
4141
↓
4242
Backend
43+
Feasibility and cost selection over workload candidates
44+
↓
4345
Deployment Plan Compiler + sources/store + operational policy
4446
↓
4547
PrecomputePlan + QueryPlan + summary definitions
@@ -49,8 +51,9 @@ Backend
4951

5052
| Owner | Decisions |
5153
| --- | --- |
52-
| Planner logical and maintenance selection | Computation semantics, guarantees, window/retention/reuse requirements |
54+
| Planner logical and maintenance candidate construction | Computation semantics, guarantees, window/retention/reuse requirements |
5355
| Planner Physical Plan Compiler | Concrete operators, schemas, dependencies, roots, sharing and materialization frontiers |
56+
| Backend candidate selection | Deployable workload candidate, using capabilities and scoped cost inputs; shared work is costed once within that candidate |
5457
| Backend Deployment Plan Compiler | Concrete source/state bindings, stored-output identities, placement, scheduling and installation version |
5558
| Backend engines | Resolve inputs, drive execution, publish results, check actual readiness and apply installed fallback policy |
5659
| Shared physical library | Operator execution, per-run sharing, backpressure, cancellation and resource contracts |
@@ -67,10 +70,13 @@ policy rather than a minimum, the binding must preserve that policy.
6770
Planner may place a build in a query DAG or a readout before a persisted scalar
6871
output. Backend execution respects the selected graph boundaries.
6972

70-
Capabilities and scoped cost evidence flow from the backend to Planner selection.
71-
Missing support makes a candidate unavailable. Deployment compilation validates
72-
the selected realization; it does not repair an unsupported candidate by changing
73-
operators, windows or boundaries. Such changes require replanning.
73+
Planner exposes supported, semantically legal physical candidates for the workload.
74+
Backend evaluates deployment feasibility and compares their scoped costs, then
75+
selects a candidate and binds its deployment. Binding failures exclude candidates;
76+
missing costs must not silently become zero. Backend may evaluate binding while
77+
pricing candidates, but cannot change their operators, windows or boundaries.
78+
The current validation uses synthetic costs. Online resource collection and
79+
feedback-driven replanning are deferred.
7480

7581
A maintenance lifecycle is a contract associated with computation, not another
7682
operator IR. A deployment plan is an operational wrapper around Physical DAGs,
@@ -90,13 +96,25 @@ A physical graph may be embedded or referenced within the bundle; either way,
9096
its operator vocabulary and computation remain Planner-owned. The backend does
9197
not copy it into a second set of Build/Merge/Estimate node variants.
9298

99+
Two concrete decisions govern these bindings:
100+
101+
| Design question | Decision and example |
102+
| --- | --- |
103+
| What identifies the source dataset? | Tenant A's and tenant B's `KLL(latency)` have different definitions. Moving tenant A's dataset to another endpoint preserves its definition. See [source identity examples](summary-catalog-sds-architecture.md#source-identity). |
104+
| Can a new plan version reuse old state immediately? | Version 43 populates its own state even if version 42 has the same definition. Same-version restart can recover eligible records. See [recovery examples](summary-catalog-sds-architecture.md#recovery-and-plan-version-changes). |
105+
93106
Bindings attach only to declared physical boundaries:
94107

95108
```text
96109
physical input slot → concrete raw source or stored-output reference
97110
physical output → persisted output or query result
98111
```
99112

113+
Backend resolves a stable logical dataset identity before requesting Planner
114+
semantic definitions. Physical source bindings must realize that identity; changing
115+
an endpoint or replica does not change it. Binding a different dataset requires a
116+
new semantic definition, not reuse of a matching field name.
117+
100118
The compiler assigns each persisted output a `stored_output_id` within the plan
101119
version. Its writer and all readers refer to the same definition and compatible
102120
format. A `StoredOutputReference` is a binding, not a separately managed catalog
@@ -188,7 +206,7 @@ For each selected physical candidate, the compiler:
188206
Backend feasibility includes persisting the selected output type. Planner may
189207
produce scalar/result frontiers as well as sketches; this does not imply the
190208
backend supports all of them. An unsupported output is rejected or excluded
191-
through Planner feasibility selection, never silently replaced with another
209+
during Backend candidate selection, never silently replaced with another
192210
frontier.
193211

194212
A query-only candidate can build state during a query; a precompute candidate
@@ -213,10 +231,16 @@ invokes the same executor and adapts results. Both propagate cancellation and
213231
resource limits. Neither interprets logical Post-ASAP nodes at runtime.
214232

215233
Cleanup respects retention and active readers/dependent producers. Storage lookup
216-
uses installed references; it does not search for an alternative summary at
234+
uses installed references; it does not search for a substitute summary at
217235
serving time. See SDS for record eligibility and recovery requirements.
218236

219-
## 7. Alternatives and tradeoffs
237+
Initial recovery is limited to the same installed plan version. A new version
238+
populates its own state, even when definitions match the previous version. During
239+
warm-up it uses its installed fallback or unavailability policy. Cross-version
240+
state adoption is deferred; equal definitions do not authorize it. See the
241+
[SDS recovery contract](summary-catalog-sds-architecture.md#recovery-and-plan-version-changes).
242+
243+
## 7. Design choices and tradeoffs
220244

221245
Re-lowering logical nodes in the backend would duplicate physical selection and
222246
allow deployment and Planner graphs to drift. Consuming Physical DAGs avoids that

‎docs/design_docs/asapplanner-migration-plan.md‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,15 @@ can reference semantic fingerprints. Definitions derived from incomplete legacy
7171
metadata must be reconstructed from authoritative plans or rejected for rebuild;
7272
do not infer missing expressions from source and grouping alone.
7373

74+
Supply stable logical dataset identities to Planner before semantic export, and
75+
validate that concrete source bindings realize those identities. Different datasets
76+
must not acquire equal definitions merely because expressions use the same names.
77+
78+
Planner exposes physical workload candidates. Backend evaluates binding feasibility
79+
and scoped costs before selecting a deployment; it does not rewrite candidate DAGs.
80+
The initial tests inject synthetic costs. Online measurements and feedback-driven
81+
replanning remain deferred.
82+
7483
Consume the selected Physical DAGs, physical boundary identities, query
7584
associations and maintenance requirements. Replace semantic-node classification
7685
with mappings from declared input/output boundaries to deployment resources.
@@ -116,9 +125,10 @@ Migrate publishers and consumers together with pinned dependencies and matching
116125
rollback artifacts. Remove obsolete plan adapters, full-logical-DAG execution
117126
and duplicated operators after the new path passes its gates.
118127

119-
Storage payload readers remain governed by the supported format policy. Reuse
120-
across plan versions requires an explicit compatibility decision independently
121-
of a binary rollback.
128+
Storage payload readers remain governed by the supported format policy. Initial
129+
recovery supports the same installed plan version. New versions populate
130+
their own state and use their installed fallback/unavailability policy during
131+
warm-up. Cross-version state adoption is deferred independently of binary rollback.
122132

123133
## 4. Acceptance evidence
124134

@@ -132,11 +142,15 @@ Acceptance includes:
132142
- Supported query-time construction and precomputed finalized outputs follow the
133143
selected phases; unsupported output bindings fail explicitly.
134144
- Missing, overlapping, incomplete or incompatible state fails eligibility.
145+
- Dataset identity changes alter definitions; endpoint/replica changes do not.
146+
- Query branches preserve the whole-query revision fence during publication.
147+
- Same-version recovery validates bindings and completeness; new-version reads
148+
never silently adopt old state and follow warm-up failure policy.
135149
- Staging failure, cancellation, resource limits, restart and version switching
136150
preserve documented behavior.
137151
- Obsolete plans are rejected and backend builds/tests do not require Collector.
138152

139-
Trace a query from Planner selection through physical compilation, deployment
153+
Trace a query from Planner candidate construction through Backend selection, deployment
140154
binding, state publication and query execution. Verify exact operations against
141155
independent results and sketches against their supported guarantees. The design
142156
is not accepted solely because example schemas parse or unit tests pass.
@@ -155,7 +169,7 @@ Ad-hoc discovery is deferred.
155169
| Planner dependency integration (#774) | Consume the shared semantic export and propagate it from selected physical outputs into deployment compilation. | No backend expression normalization or synthetic semantic fingerprint from incomplete config fields. |
156170
| Precompute/storage integration (#763) | Persist definitions and output-scoped records; authorize writes against installed bindings and recover them consistently. | Restart retains semantic descriptions; wrong-output writes fail; replacement metadata and payload remain consistent. |
157171
| Query integration (#765) | Resolve the installed deployed output and validate definition, revision, format and coverage before invoking shared execution. | A hot-bound query never reads rebuild state; stale, missing or incompatible records take the explicit failure route. |
158-
| Acceptance PRs (#728, #742, #759) | Update fixtures and process tests for the new contract; retain existing behavioral and performance gates. | End-to-end producer → persisted definition/record → recovery → bound read, with negative identity and coverage cases. |
172+
| Acceptance PRs (#728, #742, #775) | Update fixtures and process tests for the new contract; validate individual queries and ensembles using synthetic costs. | End-to-end producer → persisted definition/record → recovery → bound read, with negative identity and coverage cases. |
159173

160174
These are implementation responsibilities and acceptance gates. PR ordering
161175
must follow actual dependency commits, not an outdated stack list.
@@ -170,4 +184,5 @@ in the same implementation PR.
170184

171185
The open shared-library integration PR is #774, replacing the already merged
172186
#770. The active order after #771 is #774 → #763 → #765 → #761 → #728
173-
→ #742 → #759; old #770 base metadata is not part of this chain.
187+
→ #742 → #775. Real-evidence work in #776, #777, #778 and #759 is deferred;
188+
old #770 base metadata is not part of this chain.

‎docs/design_docs/planner-backend-glossary.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,9 @@ in the serialized API.
1010

1111
| Term | Meaning |
1212
| --- | --- |
13-
| Selected post-ASAP DAG | Planner-selected computation graph, including summary producers, shared dependencies and query readouts. |
13+
| Selected post-ASAP DAG | The Planner-produced logical computation underlying the physical candidate selected by Backend, including summary producers, shared dependencies and query readouts. |
1414
| Physical DAG | Planner-owned concrete operators, typed input boundaries, dependencies and roots; no storage identities or placement. |
15+
| Physical candidate | A Planner-produced physical realization of a workload; Backend checks deployment feasibility and selects using scoped costs. |
1516
| Deployment plan | System instantiation of physical computation with concrete source/state bindings and operational policy. |
1617
| Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. |
1718
| `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with selected maintenance requirements for its unique reachable summary producers, plus workload and costing context. |
@@ -21,7 +22,7 @@ in the serialized API.
2122
| `QueryPlan` | Planner query Physical DAGs plus backend input bindings, query/result associations and fallback policy. |
2223
| Readout / `SummaryEstimate` | Operation that obtains a query value from summary state, such as p99 from KLL. |
2324
| Derived summary state | Stored summary state computed from existing summary states. Earlier discussion calls this a “derived materialization”; it does not require a separate catalog object. |
24-
| Exact residual | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. |
25+
| Exact computation around summaries | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. |
2526
| Exact fallback | Configured execution of the original query through an exact route when the summary plan cannot serve it. |
2627

2728
For example, merging five compatible one-minute KLL summaries and storing the
@@ -51,6 +52,7 @@ Migration of existing types and fields is covered in the
5152
| Schema / encoding | Schema describes the state structure; encoding describes how that structure is represented as bytes. |
5253
| Semantic discovery | Future Planner search for legal query rewrites over persisted definitions; distinct from fingerprint equality and record lookup. |
5354
| Deployment resolution | Backend selection of authorized stored outputs realizing a selected definition. |
55+
| Logical dataset identity | Stable semantic source identity supplied before Planner definition export; distinguishes datasets independently of endpoints or replicas. |
5456
| Definition ID | Fingerprint of a versioned canonical semantic description; different input expressions must remain distinguishable. |
5557
| Provenance | Mapping from physical plan operations back to the selected Planner computation. |
5658

@@ -68,7 +70,7 @@ reclassifying logical nodes or changing that boundary.
6870
| Retention | How long state remains available; distinct from its input range and refresh cadence. |
6971
| Readiness | Whether the required state is available with valid format and sufficient coverage/completeness for a read. Plan installation alone does not establish readiness. |
7072
| Backend capability | Declaration of supported implementation combinations: algorithm/parameters, maintenance mode, input kind, window behavior and format. |
71-
| Physical cost evidence | Scoped measurements or estimates used to compare executable alternatives; includes workload and implementation context. |
73+
| Physical cost evidence | Scoped measurements or estimates used to compare executable candidates; includes workload and implementation context. |
7274
| Compiler contract | Required inputs, outputs, validation rules and guarantees, including matching writer/reader definitions, formats, partitions and plan versions. |
7375

7476
## Compilation ownership

0 commit comments

Comments
 (0)