fix(reconciler): ms-truncate standalone CAS guard; stop auto-retrying cancelled runs - #567
Open
jonwiggins wants to merge 1 commit into
Open
fix(reconciler): ms-truncate standalone CAS guard; stop auto-retrying cancelled runs#567jonwiggins wants to merge 1 commit into
jonwiggins wants to merge 1 commit into
Conversation
… cancelled runs Two reconciler bugs in the standalone (workflow_runs) path: 1. applyStandaloneTransition CAS-guarded with a plain eq(workflowRuns.updatedAt, version). Postgres timestamptz stores microseconds while JS Dates carry milliseconds, so a row whose updated_at was last written by PG now()/defaultNow() could NEVER match the snapshot version — every executor transition from such a row was permanently stale. All CAS guards now share one ms-truncating updatedAtMatches() comparison (extracted from casUpdate, which already did this correctly). 2. Cancelling a run left it FAILED with retry budget remaining, and decideFailed cannot tell a user cancellation from an agent failure — the next reconcile pass silently flipped the cancelled run back to QUEUED and reran it. Both cancel paths (workflowService. cancelWorkflowRun and the control_intent=cancel decision) now stamp retryCount = max(retryCount, maxRetries) at cancel time so the auto-retry budget is structurally exhausted. Explicit user retry via retryWorkflowRun still works — it does not consult maxRetries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two confirmed reconciler bugs in the standalone (
workflow_runs) path.Bug 1 — standalone CAS comparison was microsecond-fragile
applyStandaloneTransitioninapps/api/src/services/reconcile-executor.tsguarded its write with a plaineq(workflowRuns.updatedAt, version), while the file'scasUpdatehelper compares with millisecond truncation (date_trunc('milliseconds', ...)).Failure scenario: Postgres
timestamptzstores microseconds; JSDateversions carry only milliseconds. Any run whoseupdated_atwas last stamped by PG'snow()/defaultNow()(rather than a JS-originated write) could never match the snapshot version — every executor transition from that row returnedstale, permanently. Auto-retries, cancel transitions, and finish transitions from such rows silently never applied until some other writer happened to stamp a ms-precisionupdated_at.Fix: extracted the ms-truncating comparison into one
updatedAtMatches()helper and made it the single comparison used by every CAS guard —casUpdate(tasks / pr_reviews / persistent_agents / workflow_runs) andapplyStandaloneTransition. The sibling repo / pr-review / persistent-agent transition applicators already went throughcasUpdateand were audited to confirm no other raw-eq guard exists.Bug 2 — the reconciler auto-retried user-cancelled runs
workflowService.cancelWorkflowRun(and the reconciler's owncontrol_intent=cancelpath inreconcile-standalone.ts) transitions a run toFAILEDwith "Cancelled by user".decideFailedcannot distinguish a user cancellation from an agent failure, so whileretryCount < maxRetriesthe next reconcile pass flipped the cancelled run back toQUEUED— a cancelled run silently reran.Fix: both cancel paths now structurally exhaust the retry budget at cancel time —
retryCount = max(retryCount, workflow.maxRetries)— sodecideFailedsees no budget and leaves the run alone. No DB migration, no fragile error-message matching. An explicit user retry viaretryWorkflowRunstill works, since it does not consultmaxRetries.How the tests prove it
apps/api/src/services/reconcile-executor.int.test.ts(real Postgres + Redis):date_trunc('milliseconds', now()) + interval '456 microseconds') is flipped: the auto-retry transition now applies from the µs-stamped row, and replaying the same action from the stale snapshot is still refused — CAS integrity preserved.FAILED(noop failed_no_retry_intent, no write, no agent job) — previously it pinned the surprisingauto_retrydecision.apps/api/src/services/workflow-service.int.test.ts: service-levelcancelWorkflowRunstampsretryCount = maxRetries, and feeding the cancelled run's real snapshot intoreconcileStandalonedecidesnoop, notauto_retry; plus a test that cancel never lowers aretryCountalready abovemaxRetries.packages/shared/src/reconcile/reconcile-standalone.test.ts: cancel action'sstatusPatchcarries the exhaustedretryCount; the post-cancel row shape is left alone bydecideFailed.Verification
cd apps/api && npx tsc --noEmit— cleanapps/api123 files / 2167 passed;packages/shared19 files / 442 passedpnpm format:check— clean