Skip to content

fix(marketplace): consume the agreement-event dead-letter queue - #189

Merged
stewartjarod merged 3 commits into
mainfrom
polylane/autofix/xs41wzfia9d7
Oct 19, 2026
Merged

stewartjarod merged 3 commits into
mainfrom
polylane/autofix/xs41wzfia9d7

Conversation

@polylane

@polylane polylane Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes: Marketplace dead-letter queue has no consumer (failed agreement events expire unprocessed)

The AWS Marketplace agreement-event pipeline had a dead-letter queue that nothing read, so an event that exhausted its SQS retries was never reprocessed and was deleted when its 14-day retention lapsed. A buyer's subscription could therefore stay incomplete with no signal a human could act on. This adds the missing dead-letter consumer, which marks the affected subscription failed so the failure surfaces instead of vanishing.

flowchart LR
  EB["EventBridge rule<br/>aws.agreement-marketplace"] --> Q["MarketplaceEventsQueue"]
  Q --> SUB["MarketplaceEventsQueueSubscriber"]
  Q -->|"3 retries exhausted"| DLQ["MarketplaceEventsDlq"]
  DLQ -.->|"before: nothing consumed it"| X["expired after 14 days"]
  DLQ -->|"new consumer"| C["marketplace-dlq-consumer<br/>marks subscription failed"]
Loading
Detection evidence

Affected: acc_04a3bad4f001h13i5u6ajeod

Why this fix

Both sibling dead-letter queues (batch and workflow) already subscribe a consumer; the marketplace one did not, so the main queue redrived into a queue no Lambda was attached to. The worker's own header names that DLQ as its only retry, and the subscription schema documents a failed status that no code ever wrote. The new consumer correlates each dead-lettered event and moves the subscription to failed, the terminal state the schema reserves for this. It deliberately does not re-enqueue, which would risk a queue/DLQ loop on a record that fails deterministically; automatic reprocessing remains an open decision recorded on the issue.

Causal chain
  • Signal (alert): Marketplace dead-letter queue has no consumer on AWS 905130073023 (failed agreement events expire unprocessed)
  • Surfacing site: AWS account 905130073023 (us-east-1) at wraps-team/wraps:infra/marketplace.ts
  • Mechanism: marketplaceDlq is declared and MarketplaceEventsQueue redrives into it (retry: 3), but no marketplaceDlq.subscribe(...) exists; messages that exhaust their SQS retries are therefore never read and are deleted when the 14-day MessageRetentionPeriod elapses.
  • Producer: Marketplace events pipeline infrastructure (missing MarketplaceEventsDlq subscription), at wraps-team/wraps:infra/marketplace.ts
  • Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: AWS Marketplace agreement/onboarding events that exhaust their SQS retries; the affected buyer's aws_marketplace_subscription row stays pending and the registration is never completed.
  • Producer evidence:
    • infra/marketplace.ts: marketplaceDlq declared and marketplaceQueue set with dlq:{queue: marketplaceDlq.arn, retry:3}; only marketplaceQueue.subscribe(...) is called, no marketplaceDlq.subscribe(...)
    • infra/queues.ts: batchDlq.subscribe(...) and workflowDlq.subscribe(...) both exist, so the sibling DLQs have consumers
    • apps/api/src/workers/marketplace-events.ts header: 'Its DLQ is the only retry, and a record it cannot process leaves a buyer stuck at pending'
    • packages/db/src/schema/aws-marketplace.ts: status documented as 'pending | active | unsubscribed | failed' with no writer of 'failed'
    • Live AWS reads 2026-09-30: Lambda ListEventSourceMappings has no mapping for wraps-production-MarketplaceEventsDlqQueue-sbwfards; DLQ depth 0; MessageRetentionPeriod 1209600
5 files changed (+492/-128)
  • apps/api/src/services/marketplace-event.ts: added, +140/-0
  • apps/api/src/workers/__tests__/marketplace-dlq-consumer.test.ts: added, +163/-0
  • apps/api/src/workers/marketplace-dlq-consumer.ts: added, +142/-0
  • apps/api/src/workers/marketplace-events.ts: modified, +10/-128
  • infra/marketplace.ts: modified, +37/-0

View thread View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment.

@polylane polylane Bot added polylane severity:medium Polylane autofix severity: medium labels Sep 30, 2026
@vercel

vercel Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
wraps-web Skipped Skipped Oct 19, 2026 5:37am UTC
wraps-website Skipped Skipped Oct 19, 2026 5:37am UTC

Request Review

@polylane

polylane Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor Author

This pull request is waiting on a decision: merge it if the change is still wanted, or close it if the fix is no longer needed. Either one settles it.

It was opened on 2026-09-30 and has had no new activity for a week. If nothing happens on it within a week, Polylane closes it and keeps the branch, so reopening it brings the change back exactly as it stands.

…tter queue

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
The main worker swallows per-record errors, so what dead-letters is a whole
batch whose Lambda timed out or crashed. Records earlier in that batch may
already have been applied, and failing them would mark a live subscription
failed. Guard the update on status = pending and surface settled rows to
Sentry without touching them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QZmHPbcx2nhuuDHy86YQ2
…hem (#195)

The worker caught every per-record error, logged it, and returned success,
so SQS deleted the message. A transient database error on an activation
never reached the retries or the dead-letter queue, and the buyer stayed
pending. Report failed records as batch item failures (partialResponses is
already on) so SQS retries just those and dead-letters them once spent.


Claude-Session: https://claude.ai/code/session_014QZmHPbcx2nhuuDHy86YQ2

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@vercel
vercel Bot temporarily deployed to Preview – wraps-web October 19, 2026 05:37 Inactive
@vercel
vercel Bot temporarily deployed to Preview – wraps-website October 19, 2026 05:37 Inactive
@stewartjarod
stewartjarod merged commit 15b8000 into main Oct 19, 2026
18 checks passed
@stewartjarod
stewartjarod deleted the polylane/autofix/xs41wzfia9d7 branch October 19, 2026 05:43

This branch was previously deployed

2 inactive deployments
Preview – wraps-web — a985eb81 Deployed Oct 19, 2026 by vercel[bot]
Preview – wraps-website — a985eb81 Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant