Skip to content

fix(tasks): permanent failure on missing secrets; post-transition row from POST /api/tasks - #569

Merged
jonwiggins merged 2 commits into
mainfrom
fix/task-provisioning-and-create-response
Aug 8, 2026
Merged

fix(tasks): permanent failure on missing secrets; post-transition row from POST /api/tasks#569
jonwiggins merged 2 commits into
mainfrom
fix/task-provisioning-and-create-response

Conversation

@jonwiggins

Copy link
Copy Markdown
Owner

Two confirmed task-pipeline bugs, one fix each, both covered by tests.

Bug 1: missing agent secret made repo tasks retry provisioning forever

Failure scenario. Create a repo task (agentType claude-code) with no ANTHROPIC_API_KEY secret stored. The task-worker's provisioning path throws Secret not found: ANTHROPIC_API_KEY (scope: global) (from secret-service.retrieveSecret() via resolveSecretsForTask). The provisioning catch in task-worker.ts consults classifyError(), which marked Secret not found as retryable: true, so the task took the recoverable branch: back to queued with a 30s-delay requeue. The worker's own 3-retry cap never engaged in practice because the reconciler's applyRequeueForAgent re-enqueues queued tasks as bare { taskId } jobs, resetting provisioningRetryCount to 0 every cycle. Net effect: the task bounced queued <-> provisioning indefinitely with no terminal state and no actionable surface.

Fix. packages/shared/src/error-classifier.ts: the Secret not found: NAME pattern (the exact message shape retrieveSecret() throws, nothing broader) is now retryable: false. The provisioning catch already treats non-retryable classifications as permanent, so the first failure lands the task in terminal failed via provisioning_permanent_failure, with the clear Secret not found: ... message on errorMessage and an actionable remedy in the classified error. Other auth patterns (expired OAuth token, rate limits, etc.) keep their existing retry semantics.

Bug 2: POST /api/tasks returned the pre-transition row

Failure scenario. The repo-task branch of POST /api/tasks inserts the task (pending), transitions it to queued (or waiting_on_deps), enqueues the BullMQ job — and then responded with the stale insert row: state: "pending". Clients immediately saw a state the task was no longer in; the e2e suite had even pinned this with a comment ("the response carries the ORIGINAL createTask() row").

Fix. apps/api/src/routes/tasks.ts: respond with the row returned by taskService.transitionTask() (it already returns the post-transition row — no extra query). The response now reports queued / waiting_on_deps. The Zod response schema (task: z.record(z.unknown())) is unaffected; the OpenAPI description notes the response carries the post-transition row.

Tests

  • New e2e apps/api/e2e/provisioning-no-secret.e2e.test.ts: boots the real API server with no secrets seeded, creates a repo task, and proves it reaches terminal failed with Secret not found: ANTHROPIC_API_KEY in errorMessage, exactly one worker_pickup, one provisioning_permanent_failure event (provisioning -> failed), zero provisioning_retry events — and stays that way after several reconciler/stall-check cycles (no resurrection loop). Also asserts the POST response state is queued (bug 2 in the same server).
  • Updated e2e apps/api/e2e/repo-task.e2e.test.ts: the pinned expect(body.task.state).toBe("pending") assertion now asserts "queued".
  • Unit packages/shared/src/error-classifier.test.ts: missing-secret classifications now asserted non-retryable.
  • Unit apps/api/src/routes/tasks.test.ts: transitionTask mocks return the post-transition rows; response state asserted for both the queued and waiting_on_deps paths.

Verification

  • apps/api and packages/shared tsc --noEmit: clean
  • apps/api unit suite: 123 files / 2167 tests pass; packages/shared: 19 files / 440 tests pass (full workspace turbo test also ran green in the pre-push hook)
  • Both touched e2e files run twice back-to-back against the real-infra harness: 5/5 pass both runs, no flake
  • pnpm format:check: clean

…issing

A repo task whose agent secret was never stored (e.g. claude-code with no
ANTHROPIC_API_KEY) threw "Secret not found: NAME (scope: global)" during
provisioning, which the error classifier marked retryable. The task-worker
re-queued it on a 30s delay, and because the reconciler re-enqueues queued
tasks without the provisioningRetryCount job field, the 3-retry cap reset
every cycle: the task bounced between queued and provisioning forever with
no terminal state and no actionable surface.

Classify the exact "Secret not found: NAME" shape thrown by
secret-service.retrieveSecret() as non-retryable, so the provisioning
catch takes its permanent-failure branch: one worker pickup, a
provisioning_permanent_failure transition, and terminal failed with the
actionable message on the task row. The classification stays narrow;
other auth errors keep their existing retry semantics.

Covered by a dedicated e2e (provisioning-no-secret.e2e.test.ts) that boots
the real API server with no secrets seeded and proves the task lands in
failed with no provisioning_retry events, staying failed across
reconciler cycles.
The repo-task branch of POST /api/tasks inserted the task, transitioned it
to queued (or waiting_on_deps), enqueued the BullMQ job, and then responded
with the stale row from createTask() - state "pending", a state the task
had already left. Clients acting on the response immediately saw the wrong
state.

transitionTask() already returns the post-transition row, so respond with
that instead of re-fetching. The e2e assertion that pinned the old
behavior (repo-task.e2e.test.ts expected "pending" with a comment noting
it was the PRE-transition row) now asserts "queued", and the route unit
tests assert the response state for both the queued and waiting_on_deps
paths.
@jonwiggins
jonwiggins merged commit 86b11b9 into main Aug 8, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant