Skip to content

fix(mongodb): a retried commit finishes the writes an interrupted one left behind - #659

Merged
oluciano merged 10 commits into
developfrom
fix/656-mongo-interrupted-success-commit
Oct 23, 2026
Merged

oluciano merged 10 commits into
developfrom
fix/656-mongo-interrupted-success-commit

Conversation

@oluciano

Copy link
Copy Markdown
Owner

Summary

Fixes #656. CommitJobResultAsync returned early when the job was already terminal, so a retry after an interrupted commit skipped the continuation release and the recurring status. It now runs those idempotent tails when the stored status matches the result outcome (Succeeded or missing job for a success, Failed for a final failure).

Testing

  • Red first: commit 1 adds 6 tests, 4 failed against the old code.
  • 170 MongoStorageProviderTests pass, Release build 0 errors.
  • N1: children released (kept and purged job), recurring status (success and failure). N2: a success on a job stored as Failed releases nothing. N3: three commits with no children and no recurring job.

Checked, no change needed

  • ApplyRetryAsync: leaves the job Scheduled (not terminal), one write, the early return never applies.
  • ApplyFailureAsync: same gap, fixed here. Dead-letter runs in the executor, not in storage.
  • PostgreSQL, SQL Server, Redis commit atomically; InMemory has no partial write.

Risk and rollback

  • Risk: medium (storage, Deep). A late retry after a newer run of the same recurring job can put the older status back; documented, not guarded.
  • How to revert: revert the merge; no schema change.

@oluciano

Copy link
Copy Markdown
Owner Author

[review] 3 findings: 1 Important, 1 Nit, 1 Pre-existing (Deep, per REVIEW.md)

  1. Important — src/NexJob.MongoDB/MongoStorageProvider.cs (FinishInterruptedCommitAsync, ApplySuccessTailAsync, ApplyFailureTailAsync): a late duplicate commit can overwrite the recurring job's LastExecutionStatus written by a newer run. Disputed / documented, not guarded: the document has no per-run marker to guard on, and the retry follows the failed write within seconds. Documented in a code comment and in delivery-guarantees.md; the wiki/CHANGELOG no longer call the replay always safe. Open for the owner to decide whether a guard is worth a schema change.
  2. Nit — MongoStorageProvider.cs (result.Succeeded && storedStatus is null or ...): precedence was correct but easy to misread. Fixed: parenthesized.
  3. Pre-existing — ApplyRetryAsync / ApplyFailureAsync: the retry path has no early-return problem (job stays Scheduled, one write); the failure path had the same gap and is fixed here. Dead-letter is in the executor, outside storage. Stated in the PR body.

No weakened tests: 117 lines added, none removed.

@oluciano

Copy link
Copy Markdown
Owner Author

[review] Follow-up on finding 1 (Important): fixed, not just documented. The earlier claim that a guard needed a schema change was wrong: JobDocument already stores RecurringJobId. The tail now skips the recurring-status write when a newer job of the same recurring job exists (HasNewerRunAsync). Red test first (commit test(mongodb): a late retry of an older run...), green after; 171 Mongo tests pass. One case stays unguarded and is stated in delivery-guarantees.md: a job removed by PurgeOnSuccess has no CreatedAt to compare.


// The recurring status belongs to the latest run: a retry that arrives after a newer run of the same
// recurring job must not put the older run's status back. A purged job has no CreatedAt to compare.
var recurringJobId = await HasNewerRunAsync(storedJob, result.RecurringJobId, ct).ConfigureAwait(false)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] Important — a late duplicate commit could overwrite the recurring job's LastExecutionStatus written by a newer run.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 37911b0 (guard in HasNewerRunAsync, red test then green, 171 Mongo tests pass). My first answer, that it needed a schema change, was wrong: JobDocument already stores RecurringJobId. Not guarded: a job removed by PurgeOnSuccess, which has no CreatedAt to compare; stated in delivery-guarantees.md.

: result.RecurringJobId;

// A purged success leaves no document, so a missing job counts as Succeeded.
if (result.Succeeded && (storedStatus is null or JobStatus.Succeeded))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] Nit — result.Succeeded && storedStatus is null or ... had correct precedence but was easy to misread.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: the condition is parenthesized.

await ApplyFailureTailAsync(result, result.RecurringJobId, ct).ConfigureAwait(false);
}

private async Task ApplyFailureTailAsync(JobExecutionResult result, string? recurringJobId, CancellationToken ct)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[review] Pre-existing — ApplyRetryAsync / ApplyFailureAsync: does the same early-return gap exist outside the success path?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved, no change needed for the retry path: ApplyRetryAsync leaves the job Scheduled (not terminal) with one write, so the early return never applies. ApplyFailureAsync had the same gap and is fixed here (ApplyFailureTailAsync). Dead-letter runs in the executor, outside storage.

@sonarqubecloud

Copy link
Copy Markdown

@oluciano
oluciano merged commit fe85d35 into develop Oct 23, 2026
38 checks passed
@oluciano
oluciano deleted the fix/656-mongo-interrupted-success-commit branch October 23, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(mongodb): an interrupted success commit leaves the continuations of the job AwaitingContinuation forever

1 participant