Skip to content

A delivery to a remote that fails after its crank commits is lost with nobody told #1125

Open

Description

@sirtimid

#1064 closed the case where a remote refuses a GC delivery: #1099 aborts the crank, which restores the action and the c-list entries together, and holds that remote's actions back until the run loop next goes idle. This is the same damage reached by the other route — the delivery succeeds, the crank commits, and the send fails afterwards — where that mechanism cannot apply because there is no crank left to abort.

The sequence

  1. processGCActionSet selects an action and calls storage.setGCActions(allActionsSet) before returning it, so the durable set has already dropped it.
  2. RemoteHandle.#deliverToPeer writes the message to remotePending.<remoteId>.<seq> and returns afterCommit. Nothing has failed yet.
  3. The crank commits. The GC action is now gone for good.
  4. afterCommit hands the message to the transport and sendRemoteMessage rejects with a terminal error — PeerRestartedError, IntentionalCloseError or NetworkStoppedError (three transient failures have already been retried).
  5. #rejectAllPending advances startSeq past the message, so it is never retransmitted, and #onGiveUp fires.

Why nobody is told

RemoteManager.#handleRemoteGiveUp rejects the promises it finds via getPromisesByDecider(remoteId), which filters on kp.state === 'unresolved' && kp.decider === decider — promises the remote owes us. Two kinds of outbound delivery are invisible to it:

  • A GC action (dropExports, retireExports, retireImports) has no promise at all.
  • A notify carries a resolution for a promise the peer imported. We decided it, so it is resolved, and its decider is not the remote. Neither test matches.

So both are dropped silently.

The damage

For a GC action, exactly what #1064 describes. The kernel releases its own side before telling the endpoint, so after this the peer still holds the eref while the kernel has forgotten it. The peer's next message naming that object goes through exportFromEndpoint and mints a second kref for the same object. An incarnation change reconciles a dropped or retired export, but not a retireImports: forgetEndpointImports keeps only entries whose direction is export.

For a notify, arguably worse: the peer's imported promise never settles and nothing times it out, so whatever is waiting on it hangs for the life of that incarnation.

Reachability

Narrow but real. It needs a peer dead enough to exhaust three retries over ~30s, and to come back without restarting — a network drop that recovers, rather than a peer restart. A peer that restarts is cleaned up by the incarnation-change path; a peer that never returns makes the drift moot. The window in between is where this bites, and giveUp does not remove the handle, so the kernel will talk to that peer again.

Directions

Neither of these is the unbounded retry the original comment in #deliverGCAction rightly rejects:

  1. Make the failure restartable. #rejectAllPending currently abandons the payload. It could instead put the GC action back in the durable set, which would need the action to be recoverable from the pending message — and would have to happen inside a crank, which is the same "only the run loop writes the store" problem the transport's failure handling already has (see the Not fixed here section of feat(ocap-kernel): send a remote delivery once its crank has committed #1101).
  2. Record the orphan for the next incarnation change to sweep, which is option 2 from A remote's retireImports that fails is never reconciled #1064: either forgetEndpointImports stops filtering on direction, or a separate record covers the entries it does not.

The notify half may want a different answer from the GC half, since there is no action to restore — only a peer-side promise that will never settle.

Related

Activity

  1. sirtimid commented on Oct 21, 2026

    @sirtimid
    ContributorAuthor

    Blocked by #1152, which moves the give-up's writes (#rejectAllPending) into a crank of its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions