Build hosted realtime collaboration foundations

[callumalpass]

Status

Draft stacked on #317. Implements the provider-neutral CRDT profile, secure hosted transport foundations, and a private experimental shared browser room client through Phase 9. Collaboration remains disabled by default (MDBASE_CONNECT_HOSTED_COLLABORATION_ENABLED=false), unadvertised in readiness, absent from stable public SDK exports, and not yet integrated into the Editor.

Implemented

• ADR defining exact RecordDocument.body, LF-only admission, stable record rooms, epochs, and durable authority ownership
• pinned Yjs 13.6.32 / Yrs 0.26.0 profile cores with Rust/TypeScript fixtures, UTF-16 deltas, compaction/state-vector proofs, convergence schedules, and CodeMirror/Chromium adapter evidence
• v2 optional records.collaborate, v6 signed authorization, and exact opt-in membership/grant/replica capability ceilings while preserving prior contracts
• encrypted PostgreSQL room snapshots, updates, receipts, tickets, ciphertext-derived quotas, compaction, corruption fencing, and crash-atomic shared record materialization
• short-lived origin-bound one-shot HTTP tickets and bounded binary WebSocket transport behind the default-off feature flag
• per-batch and periodic session reauthorization, token-fingerprint binding, commit-before-Ack/fanout, idempotent replay, state-vector reconnect, strict frame/profile/mode/epoch validation, and safe oversized-state rejection
• durable per-record epoch fences that survive deletion, reject stale rooms, retire on ordinary body changes/deletes, and preserve rooms across path/frontmatter-only writes
• transactional metadata-only PostgreSQL notices plus durable ordered multi-instance catch-up, compaction fallback, epoch fencing, origin-echo suppression, listener reconciliation, and bounded workers
• graceful drain and bounded socket/database admission with local and cross-instance revocation, rotation, downgrade, and suspension convergence
• strict process-local awareness: complete sanitized snapshots, generic non-PII Participant N identities, collection/user-derived palette colors, post-Sync membership, bounded room/replica caps, sliding rate limits, TTL hide/reappearance semantics, drain linearization, and no SQL/outbox/log persistence
• immutable migration 0044 plus follow-up migration 0045 to sanitize any private-build profile names and enforce the generic-name invariant; deployment ledger/cutover/rollback checks pinned through 45
• private symbol-keyed ticket bridge in @mdbase-dev/connect with exact current-authorization, proof, scope, capability, epoch, URL, no-store, and TOCTOU validation; stable package exports remain unchanged
• private @mdbase-dev/connect-collaboration hosted room provider owning Y.Doc/Y.Text, strict binary protocol, bounded durable-Ack queue, epoch-bound reconnect, handshake/watchdog/backoff, heartbeat, TTL awareness refresh, immutable snapshots, flush, and cleanup
• LAB-only Editor manifest/build kill switch plus a dynamically imported room gateway; flag-off builds retain manifest v1 and exclude the collaboration/Yjs room chunk
• LAB Editor CodeMirror integration with exact Y.Text body ownership, room undo/awareness, read-only and reconnect gating, durable navigation flush, revision refresh before CAS mutations, body-preserving frontmatter patches, and terminal fail-closed behavior
• sequential PostgreSQL acceptance suites through Phase 8 wired into hosted-provider CI

Validation

• cargo test --locked -p mdbase-connect-hosted-provider --lib — 145 passed, 11 ignored (environment-gated)
• cargo test --locked -p mdbase-connect-protocol collaboration_awareness — 11 passed
• cargo clippy --locked -p mdbase-connect-hosted-provider --all-targets -- -D warnings
• cargo fmt --all -- --check
• pnpm --filter @mdbase-dev/connect-protocol build/test/typecheck — 56 tests passed
• pnpm --filter @mdbase/connect-server test/typecheck — 372 tests passed
• pnpm --filter @mdbase-dev/connect test/typecheck/test:public-api — 275 tests; root, advanced, crypto, packed, and browser API boundaries unchanged
• pnpm --filter @mdbase-dev/connect-collaboration build/typecheck/test — 27 tests passed
• pnpm check:architecture and pnpm check:workspace-pins
• browser SDK bundle remains below its hard ceiling; review-threshold increase is reported for the private ticket bridge
• sequential disposable PostgreSQL 18: Phase 3 batch/two-client transport, Phase 5 conventional-writer reconciliation, Phase 6 notice/multi-instance catch-up, Phase 7 drain/revocation/admission, and all Phase 8 awareness/migration scenarios
• independent adversarial reviews of runtime, authorization, privacy, migration, ticket, and room lifecycle boundaries; release-blocking findings fixed

Remaining gates

1. run real two-browser Chromium and LAB acceptance for convergence, reconnects, restarts, conventional writes, quotas, revocation, awareness, navigation flush, and drain
2. complete operational metrics/retention/benchmark, browser-bundle review, and fenced migration-0045 rollout evidence
3. refine the private API from Editor usage over several weeks before exposing or documenting any stable public SDK surface
4. keep collaboration disabled and readiness unadvertised until every rollout gate passes

LAB acceptance progress

• Merged current product main into the collaboration branch before continuing acceptance, retaining beta.86 signup/isolation changes and resolving the combined control-plane migration ledger and Cargo locks.
• Deployed independently verified signed Connect and hosted-provider candidate images to isolated LAB. Control-plane readiness, provider readiness, migration ledger 45, and the LAB-only collaboration flag remain healthy.
• Provisioned a temporary second ordinary LAB acceptance account through the merged guarded manual-invitation workflow, stored credentials in Secret Service, and restored registration to closed immediately after signup.
• Created one uniquely named run-owned hosted collection, added the second account as an editor with an explicit collaboration policy ceiling, and authorized two independent persistent browser profiles.
• Confirmed true two-user bidirectional convergence, exact baseline restoration, simultaneous and sequential room opening, reload persistence, navigation flush, and recovery across an exact hosted-provider restart. No console errors or data loss were observed in the passing runs.
• Live testing exposed policy-close churn during replica credential rotation. 8805db67 adds bounded shared-provider recovery: at most three fresh-ticket attempts, one combined handshake/policy budget, same-epoch pinning, pending-update retention, fail-closed authorization errors, and a continuously connected 30-second rearm gate. LAB then passed convergence and reload while transparently recovering the observed 1008 policy closes.
• dc8a2d56 makes experimental LAB sharing create and change member policies with collaboration enabled while flag-off builds retain the ordinary sharing payload.
• Membership downgrade testing confirmed the affected collaborator became non-editable while the owner remained Live. The role was restored afterward.
• A human LAB pass exposed a rapid-typing failure: the awareness sender ran at the provider’s exact eight-per-second rejection boundary, durable Yjs envelopes were emitted per transaction, and terminal binding removal remounted CodeMirror. 82bb55f2–b04b7c93 add a 250 ms awareness safety cadence, merge unsent Yjs updates, cap durable sends at 25/second, preserve immutable replay envelopes across reconnect, bound oversized coalesced envelopes, and retain the live editor key after fail-closed termination.
• Deployed the exact b04b7c93 Editor locally to the LAB candidate alias. LAB then passed 30- and 100-keystroke bursts, two-profile 100-keystroke bidirectional convergence, long-document scroll/focus stability, exact baseline restoration, and reload persistence. The status returned to exact Live; no terminal state, focus loss, page scroll, console error, or lost text was observed.
• Remaining UX gaps: the current Editor manifest requires write capabilities, so a viewer cannot authorize the Editor even though the room/provider support read-only collaboration; sanitized awareness is exchanged but remote participant labels/cursors are not yet rendered in CodeMirror. These remain promotion blockers, not privacy failures.
• The run-owned fixture and both isolated profiles are retained for continued acceptance. No pre-existing collection was mutated.

[callumalpass]

Phase 3 progress pushed through db6c7ed0:

• encrypted production collaboration documents/updates/receipts/tickets with exact collection/account ciphertext quota accounting;
• replay fencing, sequence continuity, bounded compaction, old-vector-preserving snapshots, and repair epochs;
• shared ordinary hosted write-set committer for records/versions/changes/head/projections/outbox;
• crate-private authorization-revalidated collaboration batch transaction with no-op/idempotent handling;
• executable PostgreSQL 18 migration/quota/ticket tests and retained Phase 0 crash proof;
• real Chromium two-CodeMirror adapter smoke test, including Unicode, disconnected state-vector convergence, duplicate/reordered updates, and local undo isolation; production bundle excludes the harness.

Collaboration remains deliberately unadvertised and unreachable from HTTP/WebSockets. Phase 4 ticket exchange, transport, post-commit delivery/revocation disconnect, headless two-client conformance, and then Editor live mode remain required before end-to-end collaboration can be tested or enabled.

[callumalpass]

Additional adversarial LAB campaign

Current Editor head: c73dd140 (candidate-b locally deployed). Hosted-provider image source: 85bb4b30, deployed to LAB by exact digest sha256:2565bc70a7a16807c3d74da0419754abe3244cf11b6212c8e8d7c1af4bb09a90 after signed candidate workflow 32843245030.

Confirmed and fixed:

• Reproduced Live editing unavailable after four note switches with fresh ticket 201s and four policy closes. The awareness registry's four-session cap was terminating the durable room while superseded browser sockets were still draining. Awareness admission limits now degrade to observer-only presence and never close an authorized durable session. A PostgreSQL regression covers a fifth same-replica room remaining durably synchronized.
• Replaced the retained React-key workaround with a CodeMirror collaboration compartment. Room bindings now install/remove dynamically without remounting the editor, fixing reopened-note stale bindings while preserving focus/scroll and detaching terminal Yjs observers.
• Added sanitized remote caret, selection, generic participant-label, and eight-color rendering. Local awareness is bounded to the protocol's four-selection limit.
• Presence labels initially polluted .cm-line DOM text. Labels now render from a CSS pseudo-element backed by a sanitized data-presence-label, leaving document text and test/copy surfaces clean.
• A long collection name pushed the New note control beneath the editor pane. Notes/type panes and list headers are now width-bounded; a Playwright hit-test regression verifies the action remains clickable.
• Authorization-driven collection changes could start the new collection while retaining old NoteSessionStore and recovery state. The workspace now clears before starting a changed collection; ordering has a deterministic hook regression.

Live evidence after fixes:

• 20/20 alternating note switches: every switch reached fresh Connecting live editing -> Live, 21/21 ticket POSTs returned 201, no terminal state.
• Two genuinely distinct persistent profiles: 100 rapid bidirectional characters converged, focus remained Note body, no socket closes, exact baseline restored, reload persisted the restoration.
• Two-window presence: generic labels only (Participant N), remote carets and range selections rendered, both rooms stayed Live, and CodeMirror line text remained exact/clean.

Qualification:

• Editor collaboration-enabled: 48 files / 312 tests passed.
• Editor default flag-off: 46 files passed, 2 skipped; 302 passed, 10 skipped.
• Collaboration package: 34 tests passed; typecheck passed.
• Hosted provider checks/tests, formatting, architecture, and diff checks passed.
• Full GitHub Editor/Desktop/Server qualification passed for 85bb4b30; later Editor-only commits are running their normal qualification.

Blocked/inconclusive: automated member editor->viewer acceptance could not reach the owner management surface because the ordinary browser session entered the LAB login redirect flow. The role mutation was never issued, so membership remained unchanged. Read-only Editor acceptance remains open.

[callumalpass]

Same-site LAB and two-account presence acceptance

Current head: ac932155.

LAB origin migration

• Connect: https://connect-lab.mdbase.dev
• Editor: https://editor-lab.mdbase.dev
• Render platform origin remains https://mdbase-connect-lab.onrender.com.
• Dedicated Pages project: mdbase-editor-lab, production branch candidate-b.
• mdbase-env lab doctor/status/login pass against the new custom Connect origin.
• Cookie-authenticated Editor management now returns /v1/me 200 with no login redirect loop.
• Connect management origin was reduced to the exact custom Editor origin after acceptance.
• cloud-ops origin registry shipped in PR mdbase-cloud-ops#259.

Confirmed presence defects and fixes

• Provider snapshots now omit the recipient socket's own process-local session before serialization. The v1 sanitized participant wire shape is unchanged; no stable identity or self marker is exposed.
• CodeMirror maps existing remote marks/widgets through each document ChangeDesc instead of rebuilding stale absolute offsets on every edit.
• Semantically unchanged awareness snapshots retain participant-array identity, so React does not replace optimistically mapped decorations with unchanged stale offsets.
• Added deterministic recipient-exclusion, equality-collision, insert/delete mapping, structural-sharing, and PostgreSQL awareness regressions.

Readiness-order defect found during two-account setup

Hosted collaboration capability grants depended on whether /ready had been called since Connect process start because HostedProviderClient.collaborationSupport() was populated only by ready(), while Render probes /health. This produced conventional Saved sessions and null collaboration_capability immediately after restart despite a capable provider.

ac932155 now calls provider.ready() on the hosted application approval path before planning the grant. A regression asserts readiness precedes replica enrollment. After deploying the exact qualified image and restarting Connect, fresh authorizations received collaboration tickets without any manual /ready priming.

Exact deployments

• Hosted provider from 1845e213: sha256:2eb6857850d19e9fb623b2ecf50af4e2fc5e5412c071b044163523cc1d412df6
• Connect from ac932155: sha256:39648640c6267f7977f4950e21863bcc42f5bb0f61e0673c18c990f54eef5c21
• Editor from 1845e213: exact Pages deployment 47626649.mdbase-editor-lab.pages.dev, served through editor-lab.mdbase.dev

Qualification and LAB evidence

• Server CI fully passed for both 1845e213 and ac932155.
• Collaboration package: 34/34.
• Editor collaboration-enabled: 313/313; flag-off: 303 passed, 10 expected skips.
• Hosted-provider library: 147 passed, 17 environment skips.
• Six ignored PostgreSQL phase-8 awareness/cap tests passed against a disposable PostgreSQL container.
• Typechecks, formatting, architecture, bundle, and CSP checks passed.
• Provisioned and registered a second ordinary LAB collaboration acceptance account (cloud-ops PR Release 0.1.0-beta.70 #263), shared a fresh [test] hosted collection as editor, and authorized both distinct profiles.
• Subagent live test: A alone rendered zero remote labels; with B joined each rendered exactly one generic participant. A-side DOM Range measurement of B's caret moved 29 -> 32 -> 29 for immediate local insert/delete (+3/-3) before awareness round trip. Exact baseline remained Live and persisted after B reload.
• Independent two-account test passed bidirectional A->B and B->A convergence, exact baseline restoration, reload persistence, and 201 collaboration tickets. Its broad presence selector counted the caret plus selection as two nodes, so that sub-check was a test-selector defect rather than a product failure; the dedicated exact-label/caret test above passed.

[callumalpass]

Follow-up adversarial presence review

A late independent review found one residual edge: a semantically changed awareness snapshot could reconfigure CodeMirror from raw absolute offsets while local durable updates were still pending, replacing the correctly mapped decoration. e62b1d81 now defers changed presence replacement while pendingUpdates > 0, preserves already mapped decorations, applies the latest replacement after the queue drains, and clears immediately on room changes. A deterministic regression covers local insertion plus a changed server snapshot during the pending interval.

The same commit also makes the Tokio watch borrow drop explicit before acquiring the awareness-room mutex/sending, and documents v1 awareness snapshots as recipient-relative in both protocol implementations. Collaboration-enabled Editor tests are now 314/314; protocol tests 56/56; Editor/protocol typechecks, Rust check/format, and architecture pass. Exact Editor deployment: e4b0da45.mdbase-editor-lab.pages.dev through editor-lab.mdbase.dev.