docs: clarify SDS source identity and version-scoped recovery

Author: zzylol

docs/design_docs/asapplanner-integration.md

@@ -35,11 +35,13 @@ Deployment at e9390031](https://github.com/ProjectASAP/ASAPPlanner/blob/e9390031
 
 ```text
 ASAPPlanner
-  Logical Post-ASAP DAG + selected Summary Maintenance Lifecycle
+  Logical Post-ASAP candidates + legal Summary Maintenance Lifecycles
       ↓ Physical Plan Compiler
-  Physical DAGs + typed input/output boundaries
+  Supported physical candidates + typed input/output boundaries
       ↓
 Backend
+  Feasibility and cost selection over workload candidates
+      ↓
   Deployment Plan Compiler + sources/store + operational policy
       ↓
   PrecomputePlan + QueryPlan + summary definitions
@@ -49,8 +51,9 @@ Backend
 
 | Owner | Decisions |
 | --- | --- |
-| Planner logical and maintenance selection | Computation semantics, guarantees, window/retention/reuse requirements |
+| Planner logical and maintenance candidate construction | Computation semantics, guarantees, window/retention/reuse requirements |
 | Planner Physical Plan Compiler | Concrete operators, schemas, dependencies, roots, sharing and materialization frontiers |
+| Backend candidate selection | Deployable workload candidate, using capabilities and scoped cost inputs; shared work is costed once within that candidate |
 | Backend Deployment Plan Compiler | Concrete source/state bindings, stored-output identities, placement, scheduling and installation version |
 | Backend engines | Resolve inputs, drive execution, publish results, check actual readiness and apply installed fallback policy |
 | Shared physical library | Operator execution, per-run sharing, backpressure, cancellation and resource contracts |
@@ -67,10 +70,13 @@ policy rather than a minimum, the binding must preserve that policy.
 Planner may place a build in a query DAG or a readout before a persisted scalar
 output. Backend execution respects the selected graph boundaries.
 
-Capabilities and scoped cost evidence flow from the backend to Planner selection.
-Missing support makes a candidate unavailable. Deployment compilation validates
-the selected realization; it does not repair an unsupported candidate by changing
-operators, windows or boundaries. Such changes require replanning.
+Planner exposes supported, semantically legal physical candidates for the workload.
+Backend evaluates deployment feasibility and compares their scoped costs, then
+selects a candidate and binds its deployment. Binding failures exclude candidates;
+missing costs must not silently become zero. Backend may evaluate binding while
+pricing candidates, but cannot change their operators, windows or boundaries.
+The current validation uses synthetic costs. Online resource collection and
+feedback-driven replanning are deferred.
 
 A maintenance lifecycle is a contract associated with computation, not another
 operator IR. A deployment plan is an operational wrapper around Physical DAGs,
@@ -97,6 +103,11 @@ physical input slot → concrete raw source or stored-output reference
 physical output     → persisted output or query result
 ```
 
+Backend resolves a stable logical dataset identity before requesting Planner
+semantic definitions. Physical source bindings must realize that identity; changing
+an endpoint or replica does not change it. Binding a different dataset requires a
+new semantic definition, not reuse of a matching field name.
+
 The compiler assigns each persisted output a `stored_output_id` within the plan
 version. Its writer and all readers refer to the same definition and compatible
 format. A `StoredOutputReference` is a binding, not a separately managed catalog
@@ -188,7 +199,7 @@ For each selected physical candidate, the compiler:
 Backend feasibility includes persisting the selected output type. Planner may
 produce scalar/result frontiers as well as sketches; this does not imply the
 backend supports all of them. An unsupported output is rejected or excluded
-through Planner feasibility selection, never silently replaced with another
+during Backend candidate selection, never silently replaced with another
 frontier.
 
 A query-only candidate can build state during a query; a precompute candidate
@@ -213,10 +224,16 @@ invokes the same executor and adapts results. Both propagate cancellation and
 resource limits. Neither interprets logical Post-ASAP nodes at runtime.
 
 Cleanup respects retention and active readers/dependent producers. Storage lookup
-uses installed references; it does not search for an alternative summary at
+uses installed references; it does not search for a substitute summary at
 serving time. See SDS for record eligibility and recovery requirements.
 
-## 7. Alternatives and tradeoffs
+Initial recovery is limited to the same installed plan version. A new version
+populates its own state, even when definitions match the previous version. During
+warm-up it uses its installed fallback or unavailability policy. Cross-version
+state adoption is deferred; equal definitions do not authorize it. See the
+[SDS recovery contract](summary-catalog-sds-architecture.md#recovery-and-plan-version-changes).
+
+## 7. Design choices and tradeoffs
 
 Re-lowering logical nodes in the backend would duplicate physical selection and
 allow deployment and Planner graphs to drift. Consuming Physical DAGs avoids that

docs/design_docs/asapplanner-migration-plan.md

@@ -71,6 +71,15 @@ can reference semantic fingerprints. Definitions derived from incomplete legacy
 metadata must be reconstructed from authoritative plans or rejected for rebuild;
 do not infer missing expressions from source and grouping alone.
 
+Supply stable logical dataset identities to Planner before semantic export, and
+validate that concrete source bindings realize those identities. Different datasets
+must not acquire equal definitions merely because expressions use the same names.
+
+Planner exposes physical workload candidates. Backend evaluates binding feasibility
+and scoped costs before selecting a deployment; it does not rewrite candidate DAGs.
+The initial tests inject synthetic costs. Online measurements and feedback-driven
+replanning remain deferred.
+
 Consume the selected Physical DAGs, physical boundary identities, query
 associations and maintenance requirements. Replace semantic-node classification
 with mappings from declared input/output boundaries to deployment resources.
@@ -116,9 +125,10 @@ Migrate publishers and consumers together with pinned dependencies and matching
 rollback artifacts. Remove obsolete plan adapters, full-logical-DAG execution
 and duplicated operators after the new path passes its gates.
 
-Storage payload readers remain governed by the supported format policy. Reuse
-across plan versions requires an explicit compatibility decision independently
-of a binary rollback.
+Storage payload readers remain governed by the supported format policy. Initial
+recovery supports the same installed plan version. New versions populate
+their own state and use their installed fallback/unavailability policy during
+warm-up. Cross-version state adoption is deferred independently of binary rollback.
 
 ## 4. Acceptance evidence
 
@@ -132,11 +142,15 @@ Acceptance includes:
 - Supported query-time construction and precomputed finalized outputs follow the
   selected phases; unsupported output bindings fail explicitly.
 - Missing, overlapping, incomplete or incompatible state fails eligibility.
+- Dataset identity changes alter definitions; endpoint/replica changes do not.
+- Query branches preserve the whole-query revision fence during publication.
+- Same-version recovery validates bindings and completeness; new-version reads
+  never silently adopt old state and follow warm-up failure policy.
 - Staging failure, cancellation, resource limits, restart and version switching
   preserve documented behavior.
 - Obsolete plans are rejected and backend builds/tests do not require Collector.
 
-Trace a query from Planner selection through physical compilation, deployment
+Trace a query from Planner candidate construction through Backend selection, deployment
 binding, state publication and query execution. Verify exact operations against
 independent results and sketches against their supported guarantees. The design
 is not accepted solely because example schemas parse or unit tests pass.
@@ -155,7 +169,7 @@ Ad-hoc discovery is deferred.
 | Planner dependency integration (#774) | Consume the shared semantic export and propagate it from selected physical outputs into deployment compilation. | No backend expression normalization or synthetic semantic fingerprint from incomplete config fields. |
 | Precompute/storage integration (#763) | Persist definitions and output-scoped records; authorize writes against installed bindings and recover them consistently. | Restart retains semantic descriptions; wrong-output writes fail; replacement metadata and payload remain consistent. |
 | Query integration (#765) | Resolve the installed deployed output and validate definition, revision, format and coverage before invoking shared execution. | A hot-bound query never reads rebuild state; stale, missing or incompatible records take the explicit failure route. |
-| Acceptance PRs (#728, #742, #759) | Update fixtures and process tests for the new contract; retain existing behavioral and performance gates. | End-to-end producer → persisted definition/record → recovery → bound read, with negative identity and coverage cases. |
+| Acceptance PRs (#728, #742, #775) | Update fixtures and process tests for the new contract; validate individual queries and ensembles using synthetic costs. | End-to-end producer → persisted definition/record → recovery → bound read, with negative identity and coverage cases. |
 
 These are implementation responsibilities and acceptance gates. PR ordering
 must follow actual dependency commits, not an outdated stack list.
@@ -170,4 +184,5 @@ in the same implementation PR.
 
 The open shared-library integration PR is #774, replacing the already merged
 #770. The active order after #771 is #774 → #763 → #765 → #761 → #728
-→ #742 → #759; old #770 base metadata is not part of this chain.
+→ #742 → #775. Real-evidence work in #776, #777, #778 and #759 is deferred;
+old #770 base metadata is not part of this chain.

docs/design_docs/planner-backend-glossary.md

@@ -10,8 +10,9 @@ in the serialized API.
 
 | Term | Meaning |
 | --- | --- |
-| Selected post-ASAP DAG | Planner-selected computation graph, including summary producers, shared dependencies and query readouts. |
+| Selected post-ASAP DAG | The Planner-produced logical computation underlying the physical candidate selected by Backend, including summary producers, shared dependencies and query readouts. |
 | Physical DAG | Planner-owned concrete operators, typed input boundaries, dependencies and roots; no storage identities or placement. |
+| Physical candidate | A Planner-produced physical realization of a workload; Backend checks deployment feasibility and selects using scoped costs. |
 | Deployment plan | System instantiation of physical computation with concrete source/state bindings and operational policy. |
 | Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. |
 | `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with selected maintenance requirements for its unique reachable summary producers, plus workload and costing context. |
@@ -21,7 +22,7 @@ in the serialized API.
 | `QueryPlan` | Planner query Physical DAGs plus backend input bindings, query/result associations and fallback policy. |
 | Readout / `SummaryEstimate` | Operation that obtains a query value from summary state, such as p99 from KLL. |
 | Derived summary state | Stored summary state computed from existing summary states. Earlier discussion calls this a “derived materialization”; it does not require a separate catalog object. |
-| Exact residual | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. |
+| Exact computation around summaries | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. |
 | Exact fallback | Configured execution of the original query through an exact route when the summary plan cannot serve it. |
 
 For example, merging five compatible one-minute KLL summaries and storing the
@@ -51,6 +52,7 @@ Migration of existing types and fields is covered in the
 | Schema / encoding | Schema describes the state structure; encoding describes how that structure is represented as bytes. |
 | Semantic discovery | Future Planner search for legal query rewrites over persisted definitions; distinct from fingerprint equality and record lookup. |
 | Deployment resolution | Backend selection of authorized stored outputs realizing a selected definition. |
+| Logical dataset identity | Stable semantic source identity supplied before Planner definition export; distinguishes datasets independently of endpoints or replicas. |
 | Definition ID | Fingerprint of a versioned canonical semantic description; different input expressions must remain distinguishable. |
 | Provenance | Mapping from physical plan operations back to the selected Planner computation. |
 
@@ -68,7 +70,7 @@ reclassifying logical nodes or changing that boundary.
 | Retention | How long state remains available; distinct from its input range and refresh cadence. |
 | Readiness | Whether the required state is available with valid format and sufficient coverage/completeness for a read. Plan installation alone does not establish readiness. |
 | Backend capability | Declaration of supported implementation combinations: algorithm/parameters, maintenance mode, input kind, window behavior and format. |
-| Physical cost evidence | Scoped measurements or estimates used to compare executable alternatives; includes workload and implementation context. |
+| Physical cost evidence | Scoped measurements or estimates used to compare executable candidates; includes workload and implementation context. |
 | Compiler contract | Required inputs, outputs, validation rules and guarantees, including matching writer/reader definitions, formats, partitions and plan versions. |
 
 ## Compilation ownership

docs/design_docs/summary-catalog-sds-architecture.md

@@ -114,6 +114,21 @@ encoding, not display strings or temporary node IDs. Its encoding and compatibil
 rules must be established before persistence; internal Planner refactoring alone
 must not force state migration. Unknown semantic versions fail validation.
 
+### Source identity
+
+Source semantics include a stable logical dataset identity, supplied by Backend
+to Planner before semantic definitions are exported. It distinguishes datasets
+and includes their semantic namespace where needed, such as tenant scope.
+`KLL(latency)` over tenant A's dataset and tenant B's dataset therefore has different
+definition IDs, even when field names and types match. An endpoint, replica or
+storage location is a deployment binding and does not change dataset identity.
+
+The authority resolving a source must preserve that identity across relocation
+and assign a different identity when the logical dataset changes. Installation
+validates that the concrete binding realizes the identity in the definition.
+Definition equality never grants cross-tenant or cross-deployment authorization.
+Future discovery must match source identity as well as expression semantics.
+
 ### Definition boundary
 
 The definition stops at the persisted output.
@@ -300,6 +315,33 @@ It does not decide whether KLL merging is semantically legal; Planner already
 made that decision. Missing or invalid state follows the installed fallback or
 unavailability policy. Plan installation alone does not establish readiness.
 
+### Consistent reads
+
+Per-record metadata/payload atomicity is necessary but insufficient. All inputs
+consumed by one QueryPlan DAG must pass the existing whole-query store-revision
+fence, including inputs on different branches. A concurrent publication that
+invalidates the fence prevents that result from being served; the installed
+failure policy applies. Preserve the
+[publication completeness contract](continuous-summary-completeness.md), including
+its conservative global fence and its distinction between accepted-input
+completeness and source event-time completeness.
+
+### Recovery and plan-version changes
+
+The initial rollout recovers records only under the same authoritative installed
+plan version and compatible bindings. Persisting definitions and payloads does
+not itself make admission metadata durable or establish exactly-once processing
+across crashes. Recovery must re-establish required eligibility; missing proof
+cannot be treated as complete input.
+
+A new plan version populates its own output namespace. Even a scheduling-only
+change with equal definition IDs does not automatically adopt the old version's
+records. Queries use the new version's installed fallback or unavailability policy
+until its state is ready. This entails rebuild work and a warm-up interval.
+In-flight runs retain their installed version, and cleanup respects active readers.
+Explicit cross-version state adoption is deferred to a separate compatibility and
+authorization design; it is not part of initial recovery or binary rollback.
+
 ## 7. Future: discovering SDS for an unregistered query
 
 The same definitions can later support queries not known when the SDS was created.
@@ -385,7 +427,7 @@ find_compatible(query)
 ```
 
 Semantic compatibility, mergeability, grouping, window composition, accuracy,
-and residual computation remain Planner decisions. Backend capability and
+and the computation over reused state remain Planner decisions. Backend capability and
 availability evidence can inform selection; a definition alone does not guarantee
 an executable deployment. Availability must be checked again at execution time.
 This extension does not require another catalog service or a new operator IR.
@@ -398,7 +440,7 @@ This extension does not require another catalog service or a new operator IR.
 3. Equal definition IDs do not make different deployed outputs interchangeable.
 4. A writer cannot publish state with a definition different from its installed binding.
 5. Runtime reads require both semantic compatibility and eligible concrete state.
-6. Bound QueryPlans directly resolve their selected outputs; they do not search for alternatives.
+6. Bound QueryPlans directly resolve their selected outputs; they do not search for substitute outputs.
 7. Ad-hoc SDS discovery happens through Planner and produces a new bound QueryPlan.
 8. SummaryStore reports available state; it never decides query rewrite legality.