OCapN port + the compiler work it forced: interop to 24/24, plus ~185 core commits

.claude/rules/prologos-syntax.md

@@ -30,6 +30,38 @@
   ```
   Body at the **same** indent as `|` is a layout violation (currently produces a hard parser error; tracked in issue #27 for diagnostic improvement). Body indent must be **strictly greater than** the `|` column.
 
+- **Never write three-deep nested `match` as a single function body.** Three-deep nested `match` inside a single function body fails to elaborate at *import time* with "Unbound variable" errors in Prologos's current elaborator. The fix is mechanical: factor into a chain of single-match helpers. Each helper does one `match`, returns to the caller, the caller does the next `match`. Same semantics for the user, no elaborator failure. OCapN Phases 15 and 21 both shipped this workaround; codified after the second occurrence. Until the underlying elaborator bug is fixed, this is the safe pattern.
+  ```
+  ;; WRONG — fails at import time
+  defn pipeline-deliver [target args v]
+    match [lookup-actor target v]
+      | some _ -> send-only target args v
+      | none ->
+          match [lookup-promise target v]
+            | none -> v
+            | some pst ->
+                match pst
+                  | pst-unresolved msgs -> ...
+                  | _ -> v
+
+  ;; RIGHT — helper chain
+  spec deliver-to-promise Nat SyrupValue PromiseState Vat -> Vat
+  defn deliver-to-promise [pid args pst v]
+    match pst | pst-unresolved msgs -> ... | _ -> v
+
+  spec deliver-to-promise-or-drop Nat SyrupValue Vat -> Vat
+  defn deliver-to-promise-or-drop [target args v]
+    match [lookup-promise target v]
+      | none -> v
+      | some pst -> deliver-to-promise target args pst v
+
+  spec pipeline-deliver Nat SyrupValue Vat -> Vat
+  defn pipeline-deliver [target args v]
+    match [lookup-actor target v]
+      | some _ -> send-only target args v
+      | none -> deliver-to-promise-or-drop target args v
+  ```
+
 ## Application style
 
 - **Uncurried** -- `defn foo [x y z] body`, `spec f A B -> C`. Multiple arguments in one bracket group.

.claude/rules/testing.md

@@ -4,7 +4,8 @@
 - **Full suite**: add `--all` flag. Also: `--jobs N`, `--timeout N`, `--no-record`, `--no-skip`
 - **Targeted tests (CRITICAL — use this, not bare `raco test`, after any production edit)**: `racket tools/run-affected-tests.rkt --tests tests/test-X.rkt --tests tests/test-Y.rkt` runs specific files with SCOPED PRECOMPILE (driver.rkt + the listed tests, ~0.5s incremental). This keeps test `.zo` linklets fresh after export changes in production modules, which `raco test FILE` alone does NOT do. Repeat `--tests FILE` per file (conventional command-line multi-arg pattern). **When to use targeted mode**: any time you've edited production code AND want to run specific tests — e.g., iterating on a failing test during development, verifying a fix hits the right paths, confirming a subset of tests before running the full suite. Avoids the class of "passes individually, fails in batch" / "reference to a variable that is not exported" (linklet mismatch) errors. Observed failure pattern: `raco make typing-propagators.rkt` + `raco test tests/test-X.rkt` → test's cached `.zo` holds a stale linklet snapshot of typing-propagators's exports; when exports changed, linklet-mismatch or silent wrong-results. The targeted runner's scoped precompile closes this gap. Origin: PPN 4C Phase 3c retrospective (2026-04-20) — the earlier sessions' runs didn't hit it because changes were internal (no export-surface changes); Phase 3c added multiple new provides and surfaced the gap.
 - **Reporting**: `racket tools/benchmark-tests.rkt --report` / `--trend FILE` / `--compare REF` / `--slowest N`
-- **Skip list**: `tests/.skip-tests` -- 2 pathological perf tests skipped by default (use `--no-skip` to include)
+- **Skip list**: `tests/.skip-tests` -- pathological perf tests + batch-worker-incompatible tests skipped by default (use `--no-skip` to include). When a test is genuinely incompatible with the batch worker (e.g., spawns subprocesses with `(exit 0)` skip-guards that kill the worker, exceeds the 120s per-file timeout for legitimate reasons), partition rather than patch the worker. The OCapN track skipped 9 tests (`test-ocapn-vat.rkt`, `test-ocapn-acceptance-l3.rkt`, plus 7 Node-spawning interop tests) and ran them via `raco test` directly in a separate CI workflow (`.github/workflows/interop.yml`). Patching the batch worker to handle subprocess-spawning would have been 1+ days of compiler infrastructure work; the partition was 30 minutes. Rule: **when test infrastructure mismatches a test's needs, partition rather than patch**; document in `.skip-tests` with the reason and the alternate runner.
+- **Microbench from Phase 1 in protocol stacks** -- protocol stacks (codecs, wire serializers, parsers over byte streams) have asymmetric latency hidden in tight inner loops. A microbench at the encode/decode boundary catches it during Phase 1 instead of Phase 9. OCapN's decoder shipped slow through Phases 1–8 because every test was a single small case; Phase 9's pipelined RPC was the first thing fast enough to expose the regression, and Phase 13 needed a tail-recursive accumulator + inline destructure for a 25× speedup. **Operational rule**: when implementing a codec/serializer/parser, every Phase 1 should include `tools/bench-ab.rkt` runs of encode + decode at varying message counts (1, 10, 100). Catching the perf bug at Phase 1 saves the multi-phase regression chain.
 - **Guideline**: Keep test files under ~20 test-cases / ~30s wall time for good thread-pool parallelism
 - **Delimiter check after .rkt edits**: Run `tools/check-parens.sh <file>` after EVERY edit to a `.rkt` file, BEFORE `raco make`. Instant (~100ms, read-only). Catches mismatched `()`, `[]`, `{}` with exact line:column. Eliminates trial-and-error bracket-balancing during compilation.
 - **Pre-compilation**: Both runners call `raco make driver.rkt` AND all test files before tests (skip with `--no-precompile`). The suite runner's `precompile-modules!` compiles BOTH — test files are NOT in driver.rkt's dependency graph, so `raco make driver.rkt` alone does NOT recompile them.

.claude/rules/workflow.md

@@ -65,3 +65,7 @@
 - **All tests green before moving on** -- never proceed to the next phase with failing tests. A failing test is a signal, not noise. Investigate immediately. If a failure is genuinely pre-existing and unrelated, document it explicitly with a tracking reference before proceeding.
 - **Update Master Roadmap on design docs and PIRs** -- writing a track design document or a PIR is a signal to update `docs/tracking/MASTER_ROADMAP.org`. Link the new design doc and/or PIR in the relevant track's row. Update track status (⬜ → 🔄 → ✅). If the design reveals new tracks or cross-dependencies, add them. The Master Roadmap is the single source of truth for all work — if it's not linked there, it's invisible.
 - **Review PIR methodology before writing a PIR** -- before writing any Post-Implementation Review, read `docs/tracking/principles/POST_IMPLEMENTATION_REVIEW.org` to refresh on the 16 questions, the template structure (§1–§9+), format principles, and anti-patterns. The Track 4 PIR was initially written as a sparse 7-section summary; reviewing the methodology revealed it was missing Bugs Found and Fixed, What Went Well/Wrong/Lucky/Surprised, Architecture Assessment, What's Next, Key Files, and cross-references to prior PIRs. A PIR written without consulting the methodology will systematically omit the evaluative and meta-learning sections that make PIRs genuinely valuable. §16 must include a longitudinal survey of the **10 most recent PIRs** (not just 2-3) — the value of longitudinal analysis is in spotting slow-moving trends.
+- **Read your own pitfalls log before each phase** -- when a track has produced a pitfalls/issues log (e.g., `2026-04-27_GOBLIN_PITFALLS.md`), grep it for syntactic constructs the next phase will use BEFORE writing code. The log is index, not just record. OCapN Phase 21 retried multi-arity defn with `nil` ctor pattern despite pitfall #18 being on file from Phase 0 — wasted ~10 minutes. Same applies to language-bug logs across tracks: scan for relevant entries during phase BARs.
+- **Codify a 2-occurrence pattern within a track immediately** -- the existing rule "patterns spanning 3+ PIRs are codification-ready" is for cross-track patterns. For *intra-track* recurrence, **2 occurrences is enough** to codify. OCapN's nested-match-import-error fix shipped at Phase 15 without being added to syntax.md or the pitfalls log; Phase 21 rediscovered the same bug and re-derived the same helper-chain workaround. When a workaround is needed twice in the same track, add it to the pitfalls log AND to the relevant rule file (e.g., `prologos-syntax.md`) immediately, not after the third occurrence.
+- **Pitfalls-log second-pass scrutiny before merge** -- when a track produces a pitfalls/issues log, schedule a "second-pass scrutiny" phase before merge. Re-test each entry against the current toolchain; delete entries that are false (out-of-scope, env-specific, first-pass diagnoses that were wrong). OCapN's pitfalls log started at 30 entries and shrank to 17 after re-testing (commits `447379a`, `b9c33ba` deleted 12 false claims). A 30-entry catalogue with 12 false entries is misleading; an 18-entry catalogue with all real entries is actionable. Tracks ending without this scrutiny pass leave noise in the project's permanent record.
+- **External cross-impl gates are mandatory test infrastructure for protocol ports** -- when a track ports a protocol against an external counterparty (e.g., `@endo/ocapn`, an XMPP server, a Matrix server), a foreign-generated golden + drift gate is mandatory Phase 1 work, not optional. Pattern: write a generator script that runs inside the foreign implementation, commit its output as a fixture, add a CI step `git diff --exit-code` on the fixture. Hand-written wire vectors are a last-resort substitute. OCapN's drift gate caught 4 of 5 interop bugs that hand-written tests would have missed. The D:I ratio target (1.5:1 longitudinal) is **conditional on external validation existence** — with an external counterparty + drift gate, low D:I is fine; without one, low D:I is technical debt.

.github/workflows/interop.yml

@@ -0,0 +1,87 @@
+name: Interop (cross-runtime Syrup byte-equality)
+
+# Runs the Phase-4 cross-impl test:
+# - regenerates the Syrup fixture from `@endo/ocapn` (the JS reference)
+# - asserts the regenerated bytes match the committed fixture (drift gate)
+# - runs the Racket cross-impl test that pins Prologos's encoder against
+#   the JS bytes
+#
+# This catches:
+#   - drift in Prologos's wire codec (existing fixture catches it)
+#   - drift in @endo/ocapn's wire format (regenerate-then-diff catches it)
+#
+# Design doc: docs/tracking/2026-04-29_OCAPN_INTEROP_DESIGN.md
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  syrup-byte-equality:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Racket
+        uses: Bogdanp/setup-racket@v1.11
+        with:
+          version: '9.0'
+
+      - name: Install Node 22
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install JS interop deps (@endo/ocapn + @endo/init)
+        run: |
+          cd tools/interop
+          npm install --no-audit --no-fund
+
+      - name: Regenerate Syrup fixture from @endo/ocapn
+        run: |
+          node tools/interop/gen-syrup-vectors.mjs > /tmp/regen.txt
+
+      - name: Drift gate — committed fixture must match @endo/ocapn output
+        run: |
+          if ! diff -u racket/prologos/tests/fixtures/syrup-cross-impl.txt /tmp/regen.txt; then
+            echo ""
+            echo "FIXTURE DRIFT — @endo/ocapn output no longer matches the"
+            echo "committed fixture. Either:"
+            echo "  - the JS reference impl changed its wire format (genuine"
+            echo "    upstream change — investigate before regenerating)"
+            echo "  - or someone bumped @endo/ocapn locally and forgot to"
+            echo "    refresh the fixture: re-run"
+            echo "    \`node tools/interop/gen-syrup-vectors.mjs > racket/prologos/tests/fixtures/syrup-cross-impl.txt\`"
+            echo "    and commit the result."
+            exit 1
+          fi
+
+      - name: Install Prologos deps
+        run: cd racket/prologos && raco pkg install --auto --skip-installed
+
+      - name: Pre-compile modules
+        run: cd racket/prologos && raco make driver.rkt
+
+      - name: Run cross-impl test (Phase 4 — static byte-equality)
+        run: cd racket/prologos && raco test tests/test-ocapn-syrup-cross-impl.rkt
+
+      - name: Run live-interop test (Phase 5 — real TCP exchange Racket↔Node)
+        run: cd racket/prologos && raco test tests/test-ocapn-live-interop.rkt
+
+      - name: Run handshake test (Phase 6 — bidirectional op:start-session)
+        run: cd racket/prologos && raco test tests/test-ocapn-handshake.rkt
+
+      - name: Run conversation test (Phase 7 — 3-frame multi-op exchange)
+        run: cd racket/prologos && raco test tests/test-ocapn-conversation.rkt
+
+      - name: Run RPC test (Phase 8 — conversational state machine)
+        run: cd racket/prologos && raco test tests/test-ocapn-rpc.rkt
+
+      - name: Run pipelined test (Phase 9 — multi-turn RPC)
+        run: cd racket/prologos && raco test tests/test-ocapn-pipelined.rkt
+
+      - name: Run abort test (Phase 10 — graceful op:abort teardown)
+        run: cd racket/prologos && raco test tests/test-ocapn-abort.rkt

.github/workflows/test.yml

@@ -24,5 +24,11 @@ jobs:
       - name: Pre-compile modules
         run: cd racket/prologos && raco make driver.rkt
 
-      - name: Run full test suite
+      - name: Run full test suite (skipped: heavy OCapN tests + interop)
         run: cd racket/prologos && racket tools/run-affected-tests.rkt --all --no-record
+
+      # Heavy OCapN tests skipped by run-affected-tests because they
+      # exceed the batch-worker's 120s per-file timeout. `raco test`
+      # has no such limit. Total wall ~5 min, well under our 30 min budget.
+      - name: Run heavy OCapN tests directly
+        run: cd racket/prologos && raco test tests/test-ocapn-vat.rkt tests/test-ocapn-acceptance-l3.rkt