fix(agents): execute same-cycle due schedules in deterministic order - #1
Open
Glavin001 wants to merge 1 commit into
Open
fix(agents): execute same-cycle due schedules in deterministic order#1Glavin001 wants to merge 1 commit into
Glavin001 wants to merge 1 commit into
Conversation
The alarm() due-schedule scan (SELECT * FROM cf_agents_schedules WHERE time <= now) had no ORDER BY, so when multiple schedule rows were due in the same alarm cycle (e.g. after a DO restart or a delayed alarm) their relative execution order was an accident of the storage engine's scan order. Interactions between same-cycle schedules were racy and unreproducible. Order the scan by (time ASC, id ASC) so earlier-due schedules always run first and ties break deterministically. Align the fiber-recovery scan over cf_agents_runs the same way (created_at ASC, id ASC). Adds a workers-pool regression test that inserts due rows in an order contradicting (time, id) order and asserts dispatch follows the deterministic order (fails without the ORDER BY). Co-Authored-By: glavin@coframe.com <glavin.wiechert@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Tested end-to-end via the workers-pool test suite ( Results
Failing-first evidence (fix reverted to main)Passing run on this branchNote: the fiber-recovery scan ordering ( Tested by Devin |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
alarm()due-schedule scan had noORDER BY:When multiple schedule rows are due in the same alarm cycle (common after a DO restart, a delayed alarm, or accumulated overdue rows), their relative execution order was an accident of the storage engine's scan order. Any interaction between same-cycle schedules — e.g. an app-level watchdog schedule racing the SDK's
_chatRecoveryContinuecontinuation — was racy and unreproducible: whichever row happened to scan first changed the outcome.This orders the scan deterministically:
so earlier-due schedules always dispatch first and same-
timeties break stably byid. The fiber-recovery scan overcf_agents_runs(_runFiberRecoveryPass) is aligned the same way (ORDER BY created_at ASC, id ASC) so recovery of orphaned runs is also processed oldest-first and reproducibly.Kept deliberately minimal (two
ORDER BYclauses) so it rebases cleanly onto theAgentSchedulerextraction in cloudflare#1897.Regression test
schedule.test.ts› "same-cycle deterministic ordering": aTestScheduleAgenthelper inserts already-due rows in an order that contradicts(time, id)order, drivesalarm()once in-instance, and asserts the callbacks executed in(time, id)order. Both cases fail without theORDER BY(dispatch follows insertion/scan order) and pass with it.Includes a patch changeset for
agents.Link to Devin session: https://app.devin.ai/sessions/1745b6a41ce947c69f12ffcd9c1dd9d9
Requested by: @Glavin001