Skip to content

feat(server): archive stale scheduled tasks and delete them 30 days later - #222

Merged
jayleaton merged 4 commits into
mainfrom
feat/schedules-cleanup-20261113
Nov 13, 2026
Merged

jayleaton merged 4 commits into
mainfrom
feat/schedules-cleanup-20261113

Conversation

@jayleaton

@jayleaton jayleaton commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

Scheduled tasks never went away: fired one-shots, paused tasks and tasks failing every run stayed in every list. This adds a reversible cleanup rule on the existing scheduler tick.

Scheduler tick (5 s) ──► "scheduled-task-cleanup" source, gated to once an hour
  └─ ScheduledTaskService.cleanupStale()
       for each row: cleanupDecision(row, {running, threadArchivedAt, latestDeliveryAt})   ← pure, table-tested
         keep    ← running, or enabled with a future run and no failures
         archive ← one-shot fired 10 d ago      → "completed"
                   paused / off for 10 d        → "paused"        (clock: updated_at, so an edit restarts it)
                   ≥3 consecutive failures over 10 d → "failing"
                   bound thread archived 10 d   → "thread_archived" (agent tasks excluded: they relaunch)
                   webhook never triggered 30 d → "webhook_unused"  (any delivery restarts the clock)
         delete  ← archived 30 d ago (row + webhook deliveries)
  • Archive = enabled = 0 + archived_at + archive_reason, guarded on updated_at so a concurrent edit or run wins. Tokens are untouched, so a restored webhook keeps its URL.
  • setEnabled(true) and any save clear the archive (and the old failure count).
  • Failure counters (consecutive_failures, failing_since) are written by markCompleted, releaseStuckRun and restart recovery; a success resets both.
  • Fork migration 23 adds the four columns and backfills thread_mode = 'continue' for profile rows. The legacy V1 import now writes continue too (it left them on new_thread).
  • Thread auto-archive fix: getAutoArchiveCandidates skips threads with an enabled bound task, which would otherwise fail every later run.
  • Clients: archivedAt / archiveReason on ScheduledTask (optional, so older clients just see a paused task). Web and mobile Settings get an Active/Archived filter with the reason, "Deletes on " and Restore (a past one-shot opens the editor). Thread panel and Agents board hide archived tasks. Shared helpers live in packages/client-runtime/src/scheduledTaskArchive.ts.
  • Docs: docs/user/project-settings.md (new "Stale tasks are archived"), docs/user/thread-sidebar.md.

Windows are fixed (10 days, 30 for unused webhooks, 30-day retention) per JJ's decisions; there is no setting.

Evidence

The 11 cases from the design, written first:

  • Before (this branch's tests against the unchanged server): 8 failing — Cannot find module './cleanup.ts' (decision table), no such column: archived_at (service, webhook, migration cases), expected 'new_thread' to equal 'continue' (legacy import), auto-archive candidates expected [ …(6) ] to deeply equal [ …(5) ].
  • After: vp test run on cleanup, ScheduledTaskService (base/webhook/profile), fork 023 migration and ProjectionSettlement: 6 files, 105 tests passed. Wider: src/scheduledTasks, ForkMigrations, Migrations, ThreadSettlementService, runtimeLayer, ProjectionSettlement: 17 files, 411 passed.
  • Time is driven by TestClock.setTime / adjust; webhook cases use the mocked scheduler so no ticks run. No sleeps.
  • vp run check:changed: format, lint, knip, typecheck (41 packages), tests beside changes (19 files) and react compiler all pass.
  • Client: scheduledTaskArchive.test.ts (9), web logic + ScheduledTasksSettings.browser.test.tsx (34) pass.

Real-client captures for the Archived filter are added below after the web pass.

Merge Danger

Door: one-way for deleted rows. Archiving is reversible; deletion 30 days after archive is not. The migration only adds columns and updates thread_mode on profile rows (which the service already forced to continue on save).

Blast Radius: service

The first sweep after deploy archives existing stale tasks (on JJ's live data, the 11-day-old fired one-shot immediately). Nothing is deleted for at least 30 days after that.

Stack: base of three. PR 2 (MCP) and PR 3 (UI unification) sit on this.

Implemented by Claude Opus 5.5 in Claude Code (T3 Code).

Captures (web dev client, seeded data)

These show the Settings list with this PR's Active/Archived toggle. #226 replaces the toggle with the full filter and fixes how it overlaps the heading at 390px.

Active Archived, with reason, deletion date and Restore

🤖 Generated with Claude Code

…ater

Scheduled tasks that stop doing anything piled up forever. An hourly sweep on
the existing scheduler tick now archives a task (turns it off and records why)
after 10 days as a fired one-time task, paused task, failing task (3+
consecutive failures over 10 days) or task whose thread was archived, and a
never-triggered webhook after 30 days. Archived tasks are deleted 30 days after
archiving, with their webhook deliveries. Re-enabling or saving restores one,
and a restored webhook keeps its URL.

Fork migration 23 adds the archive and failure-counter columns and sets agent
tasks that legacy imports left on new_thread to continue. Thread auto-archive
now skips threads with an enabled bound schedule, which would otherwise fail
every later run. Web and mobile Settings get an Active/Archived filter with
the reason, the deletion date and Restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XL labels Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Baseline: 57f484f · PR result: 3be595a · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@jayleaton jayleaton left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT #222: CHANGES

Confirmed the inline finding with a deterministic service test. Reviewed head 539ab4c.

Repository checks against main passed apart from a missing local Chromium executable; after installing it, the 2 browser files passed (9 tests). The other 17 test files passed (263 tests). Format, lint, knip, typecheck and React compiler checks passed.

… started

The sweep reads task rows and active runs once. A manual run that started
while the sweep waited on an earlier row's thread lookup left archived_at
unchanged, so the delete removed the running task. The delete now requires
the row's read updated_at and a non-running status, and holds the run
reservation so a run cannot start mid-delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton

Copy link
Copy Markdown
Owner Author

Fixed in e1e6d06: the cleanup sweep no longer deletes a task whose manual run starts after its snapshot.

@jayleaton jayleaton left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT #222: CHANGES

The original regression now fails before the fix and passes after. check:changed against main passes, including lint, typecheck and all 273 focused tests. The new run reservation introduces a separate restored-webhook delivery race, reproduced before/after and described inline.

…'s delivery

Cleanup took the run reservation while deleting, so a webhook restored and
triggered in that window was refused as already running and its accepted
delivery was recorded as dispatch_failed. Cleanup now only skips tasks that
are running; the updated_at and status guards on the delete already drop it
when a run, restore or edit lands first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton

Copy link
Copy Markdown
Owner Author

Fixed in 278f960: cleanup no longer reserves the task, so a webhook restored while the sweep is about to delete it still runs its accepted delivery.

@jayleaton jayleaton left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT #222: CHANGES

The restored-webhook issue is fixed: Doug’s regression fails on e1e6d06 and passes on this head, and Randy’s previous webhook reproduction passes too. The original manual-run regression passes. Both basic orders after the in-memory running check are safe: a persisted start prevents deletion, and deletion before the start UPDATE aborts dispatch if the ID stays absent. A same-ID recreation in that last gap exposes the additional race described inline.

check:changed against main passes, including formatting/lint, knip, GraphQL, 41-package typecheck, all focused tests and React compiler checks. The isolated row-generation-guard experiment passes all five targeted checks; the proposed guard was reverted locally afterward. No commits, pushes or merges.

RANGE-DIFF #224: PURE — e1e6d06..e7d0f17 versus 278f960..b4cce2f has two '=' entries.
RANGE-DIFF #226: PURE — e7d0f17..6d40a19 versus b4cce2f..8a47c72 has three '=' entries.

A run reads the task, then marks it running by id alone. If the task was
deleted and recreated under the same id in between, as cleanup plus an
idempotent create replay can do, the old run started on the replacement and
stamped its history. The start now also matches the created_at it read and
aborts when the row was replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton

Copy link
Copy Markdown
Owner Author

Fixed in 3be595a: a run's start now matches the task's created_at, so it can't start on a task deleted and recreated under the same id.

@jayleaton jayleaton left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT #222: MERGE

Reviewed 3be595a. The start UPDATE now matches the ID and created_at read by runTask, so deletion/recreation returns “Schedule task not found” before dispatch and cannot stamp the replacement. Doug’s recreation regression fails against 278f960 and passes here. All eight targeted checks pass: the original manual-run race; both write orderings after cleanup’s running check; same-ID recreation; the earlier restored-webhook regression and Randy reproduction; and the failure-path extension.

The stuck-run argument holds under the service’s run reservation: the replacement cannot start while the old run is exiting, so it remains “never” and fails releaseStuckRun’s running-status predicate. I verified a replacement start is rejected during that window, its history remains unchanged after the old run fails, and a subsequent replacement start succeeds normally. No remaining race or new finding identified in this change.

check:changed against main passes: formatting, lint, knip, GraphQL, 41-package typecheck, 19 focused files (275 tests), and React compiler checks. All commands ran through heavy.sh. No commits, pushes or merges.

RANGE-DIFF #224: PURE — e1e6d06..e7d0f17 versus 3be595a..e3fbc4e has two '=' entries.
RANGE-DIFF #226: PURE — e7d0f17..6d40a19 versus e3fbc4e..d3303ea has three '=' entries.

@jayleaton
jayleaton merged commit b48f788 into main Nov 13, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant