connector-jcode: a production restart may strand DMs on a dead lifecycle or stale launch

[davidfarah2003]

Summary

Issue #910 included a reported restart shape that remains valid evidence but did not reproduce in the controlled path used for PR #1079.

A Jcode seat acquired a second session journal after a runtime restart. Senders kept resolving the seat from presence and every cotal_dm call reported success, but the replacement received none of those DMs. The replacement also booted from its stale launch brief instead of carrying the predecessor state.

This is separate from the reproduced active-turn stall fixed by PR #1079. That defect occurs without a restart: directed traffic stays in the connector-owned automatic queue until the current turn ends. PR #1079 routes it through Jcode soft interrupts and commits at the clean containing-turn boundary.

What did not reproduce

A controlled same-lifecycle Jcode host replacement behaved correctly:

1. the sender retained the predecessor presence record;
2. the predecessor exited;
3. the sender published a DM before successor presence;
4. the replacement reused the same wire principal and lifecycle durable;
5. the prior Harness session resumed;
6. the replacement recipient session observed the DM.

That cell does not depend on the active-turn soft-interrupt fix. It shows the reported restart loss and the active-turn stall are different mechanisms.

Remaining discriminator

Reproduce a production runtime restart that either:

• changes the DM lifecycle or wire principal while senders still resolve stale presence, or
• launches the replacement with stale material that addresses or binds the wrong lifecycle resources.

Neither occurred in the controlled replacement. The original report may still be accurate. It is unreproduced here, not disproven.

Acceptance bar

• Reproduce live before fixing.
• Assert recipient-observed delivery, not sender publish success, broker ack, presence, or process liveness.
• Prove whether the replacement bound the predecessor DM durable or a fresh lifecycle frontier.
• Record the sender-resolved principal, replacement principal, lifecycle uid, durable name/frontier, and whether a second journal was created. Do not surface argv or cmdline.

Related

• mesh delivery: a DM to a live peer can be silently dropped with a success receipt, a seat can run without ever joining, and a timed-out spawn still lands and reserves the name #910: original mixed report and evidence
• connector inbound delivery wedges while the send path stays healthy, so a seat looks alive and cannot be steered #1075: consumer wedged behind a stuck turn, which is a separate class
• PR fix(jcode): deliver directed messages into active turns #1079: reproduced no-restart active-turn stall fix

[davidfarah2003]

At exact main SHA 9c07e29df2eb51c59db05c67852b87f63d3847a3, the production restart discriminator did not reproduce and needs one coordinate-complete specimen. The broker-free Jcode session-resume suite passed 15/15: it resumes the richest matching working-directory session, and refuses foreign, archived, empty, or stub sessions. Authed connector config also requires the launcher lifecycle uid and binds the consuming endpoint's DM/delivery/history durables to it. Stronger positive control from PR #1079: a real same-lifecycle replacement retained predecessor presence, reused the wire principal and lifecycle durable, resumed the prior Harness session, and recipient-observed a DM sent before successor presence. The PR explicitly records that the production second-journal loss did not reproduce. Public #910 evidence preserves the restart hypothesis but not the required coordinate set. A private session census found relevant homes with multiple journals, but multiple journals alone cannot prove a lifecycle/principal change or stranded DM, and no message bodies, argv or secrets were inspected. Distinct from #1075; PR #1079 covers the no-restart active-turn stall. Recommend bug, triage:needs-info, area:connector, severity:high. Before fixing, capture one failure with sender-resolved principal, replacement principal, lifecycle uid, durable name/frontier, predecessor and replacement journal ids, and recipient-observed non-delivery.

[davidfarah2003]

At exact main SHA 1f7370eeec4e5c435dd1890fbdb8ef262fafed09, the reported restart loss is not reproduced.

I reconstructed the production ordering in a throwaway JWT-auth space with the real manager and real connector host; only the Harness API peer was synthetic. After the predecessor process was SIGKILLed, the sender still resolved the predecessor presence record. A DM published while manager recovery was in progress became pending on the same lifecycle-scoped DM durable. The replacement then joined with the same wire principal and lifecycle uid, a different process pid, and consumed that exact message: the durable's delivered stream sequence advanced from 0 to 1 and the replacement Harness session recipient-observed the canary. The durable name and activation start sequence were unchanged across the restart. The replacement attached the prior stored session, and the initial launch brief was dispatched once rather than replayed.

This matches the code mechanism: the manager mints a lifecycle once per spawn; supervised recovery rebuilds the launch from retained options, preserves principal, credentials, lifecycle uid, and durable binding, removes the one-time prompt, and requires a heartbeat newer than stale predecessor presence before accepting the replacement. The connector selects the richest matching stored session instead of creating a restart stub.

Corroborating isolated suites passed: smoke:jcode-mid-turn-delivery 6/6, smoke:manager-supervise-restart 21/21, smoke:jcode-lifecycle 18/18, and smoke:jcode-session-resume 15/15. The full workspace build also passed.

Classification: bug, area:connector, severity:high, triage:not-reproduced. The single still-missing fact for the historical specimen is a coordinate-complete failing restart showing either a changed replacement principal/lifecycle/durable frontier or stale launch material that differs from the current manager recovery path. Without that divergence, current main routes the transition DM to the replacement rather than a dead lifecycle or stale launch.