Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,13 @@ bunmail/
│ │ │ ├── services/
│ │ │ │ ├── webhook.service.ts ← CRUD
│ │ │ │ ├── webhook-dispatch.service.ts ← Outbox: enqueues webhook_deliveries in the caller's transaction (#30, #184)
│ │ │ │ ├── webhook-delivery.service.ts ← Claim / send / reschedule / replay (#30)
│ │ │ │ ├── webhook-delivery.service.ts ← Claim / send / reschedule / replay (#30): public surface, re-exports the webhook-delivery-*.service.ts files (#387)
│ │ │ │ ├── webhook-delivery-schedule.service.ts ← Retry backoff and attempt cap
│ │ │ │ ├── webhook-delivery-claim.service.ts ← Fair claim, attempt lease, signing secret
│ │ │ │ ├── webhook-delivery-attempt.service.ts ← One HTTP attempt behind the SSRF guard (pinned, no redirects, no decompression)
│ │ │ │ ├── webhook-delivery-query.service.ts ← List, get, replay
│ │ │ │ ├── webhook-delivery-outcome.service.ts ← Records an attempt's outcome
│ │ │ │ ├── webhook-delivery-retention.service.ts ← Retention purge
│ │ │ │ ├── webhook-delivery-worker.service.ts ← Poll loop + retention cleanup (#30)
│ │ │ │ └── replay-limit.service.ts ← Replay budget per delivery, in memory (#258)
│ │ │ ├── dtos/
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Internal: `mailer.service.ts` split by concern, no behaviour change (#380).** `sendMail` and `SendMailResult` stay in `mailer.service.ts`; the rest of the 1300-line file moved verbatim into five files beside it: `mailer-message.service.ts` (the message and its headers), `mailer-state.service.ts` (starting delivery state, signed return paths), `mailer-mta-sts.service.ts` (MTA-STS hosts and verified TLS), `mailer-delivery.service.ts` (MX sessions, throttle, rate budget, spans) and `mailer-outcome.service.ts` (recording each outcome). `mailer.service.ts` re-exports every symbol it exported before, with identical types, so no importer changes. Docs: `docs/emails.md`, `ARCHITECTURE.md`.
- **Internal: `email.service.ts` split by concern, no behaviour change (#380).** The 1106-line file moved verbatim into four files beside it: `email-create.service.ts` (`createEmail`, the idempotent send, the insert), `email-prepare.service.ts` (every check before the insert), `email-query.service.ts` (reads, lists, tag filters, attachment lookups) and `email-trash.service.ts` (trash, restore, permanent delete). `email.service.ts` re-exports every function it exported before, with identical types, so no importer changes and the e2e mocks of the module still intercept every call. Docs: `docs/emails.md`, `ARCHITECTURE.md`.
- **Internal: the two SMTP servers' services split by concern, no behaviour change (#387).** `smtp-submission.service.ts` (929 lines) keeps `start`, `stop`, `submissionMaxMessageBytes` and the server state; the rest moved verbatim into four files beside it: `smtp-submission-session.service.ts` (per-connection stats, log context, trace spans, SMTP errors), `smtp-submission-tls.service.ts` (the implicit-TLS listener, #349, #351), `smtp-submission-handlers.service.ts` (connect, AUTH, RCPT TO and close handlers) and `smtp-submission-data.service.ts` (the DATA handler). `smtp-receiver.service.ts` (838 lines) keeps `start`, `stop` and the server state and re-exports `storeInboundEmail`; the rest moved verbatim into `smtp-receiver-session.service.ts`, `smtp-receiver-handlers.service.ts` (connect, MAIL FROM, RCPT TO and close handlers), `smtp-receiver-data.service.ts` (the DATA handler and its branches) and `smtp-receiver-store.service.ts` (`storeInboundEmail`). The smtp-server handlers are unchanged and still built when `start()` runs, now by two functions per server that `start()` spreads into the `SMTPServer` options in the original order. Both modules export the same symbols with identical types, so no importer or test changes. Docs: `docs/smtp-submission.md`, `docs/inbound.md`, `docs/feedback-loops.md`, `docs/dmarc-reports.md`, `docs/suppressions.md`, `ARCHITECTURE.md`, `THREAT_MODEL.md`.
- **Internal: `webhook-delivery.service.ts` split by concern, no behaviour change (#387).** The 849-line file moved verbatim into six files beside it: `webhook-delivery-schedule.service.ts` (the retry backoff and attempt cap), `webhook-delivery-claim.service.ts` (the fair claim and the signing secret), `webhook-delivery-attempt.service.ts` (one HTTP attempt: SSRF re-validation, DNS pinning, no redirects, no decompression, the bounded body read), `webhook-delivery-query.service.ts` (list, get and replay, scoped by API key), `webhook-delivery-outcome.service.ts` (recording an attempt) and `webhook-delivery-retention.service.ts` (the retention purge). Every security check is byte-for-byte the same code. `webhook-delivery.service.ts` re-exports every symbol it exported before, with identical types, so no importer or test changes. Docs: `docs/webhooks.md`, `ARCHITECTURE.md`.
- **Feedback-report DKIM is verified by BunMail's own verifier, not mailauth (#373, part 1).** `src/modules/inbound/services/dkim-verifier.service.ts` implements RFC 6376 verification on `node:crypto` and `dns/promises`, no new dependency: rsa-sha256 with keys of at least 1024 bits and ed25519-sha256 (RFC 8463), rsa-sha1 refused (RFC 8301), simple and relaxed canonicalization, `l=`, header fields selected bottom up with over-signing, `x=` expiry, and the key record rules (`v`, `k`, `h`, `s`, `t`, an empty `p=` is a revoked key). The feedback handler's bounds are unchanged (no signature: no lookup; more than five signatures or a header section over 64 KB: skipped; 10 s cap), but only DKIM-Signature fields are counted and verified now, never ARC-Message-Signature. The logged `dkim` field changes in a few cases: a body hash mismatch is `fail` (mailauth's default mode said `neutral`), a missing, revoked or unusable key is `permerror` (was `neutral`), and signatures mailauth's default mode accepted with a warning (rsa-sha1, `i=` outside `d=`, a key record syntax error) are now `policy`, `neutral` and `permerror`. The verdict is still logged only, never required. Differential fixtures from mailauth (`test/fixtures/dkim-differential.json`) pin the agreement and each intended difference; the send-attachments integration test now checks BunMail's outgoing signature with the new verifier, and the verifier has 10 MB time-budget tests like the other inbound parsers. mailauth stays installed until the SPF part lands. Docs: `docs/feedback-loops.md`, `docs/inbound.md`, `docs/testing.md`, `ARCHITECTURE.md`, `THREAT_MODEL.md`.
- **SPF is evaluated by built-in code instead of mailauth (#373, part 2).** The SPF check of domain verification now runs BunMail's own RFC 7208 `check_host()` in strict mode (`spf-evaluator.ts` in the domains module, on `dns/promises`, no new dependency), with the same outward behaviour: sender `postmaster@<domain>`, HELO `MAIL_HOSTNAME`, a 10 s deadline, and a `temperror` shown as `unknown`. It covers every mechanism (dual CIDR on `a` and `mx`), `redirect`, macros with transformers, the limits of 10 DNS lookups, 10 names per `mx` or `ptr` answer and 2 void lookups, and treats any syntax error in the record as a `permerror`; `exp` is checked for syntax, not evaluated. Bun's `ENOTFOUND` (NXDOMAIN or no data) and `ENODATA` count as empty answers, any other DNS error as `temperror`. It passes all 203 cases of the OpenSPF RFC 7208 test suite (vendored unchanged under `test/fixtures/spf/` with its BSD-3-Clause license) and 161 BunMail cases, and agrees with mailauth's recorded results on every case but two intended differences: an `mx` answer of more than 10 names is a `permerror` even when an earlier host matches (RFC 7208 4.6.4), and a PTR answer with a trailing dot still validates. mailauth itself is removed in a later step. Docs: `docs/domains.md`, `docs/users.md`, `ARCHITECTURE.md`.
- **Test mocks are complete, and CI runs the tests in random order (#293).** Bun's `mock.module` leaks across test files, so a mock that left out an export of its module, or a field of the `config` object, broke whichever file imported that module next, depending on file order (which is why such failures showed up on CI Linux and not on macOS). Every `mock.module` in `test/` now provides every export of its target: the `config`, `logger` and `dns/promises` mocks spread the real module under their stubs, and every `config` stub starts from the real `config` object (top level and nested), so a stub only overrides values and never drops one. A new `bun run lint:mocks` check (`test/mock-completeness.ts`, no new dependency) fails CI and the pre-commit hook when a factory misses an export, so a new export fails the PR that adds it. CI also runs the unit and e2e tiers in random order with three fixed seeds (1, 42, 1337); the order-dependent failures this surfaced (the unsubscribe-token and one-click tests seeing a leaked `config` without `dkimEncryptionKey`) are fixed. Tests only. Docs: `docs/testing.md`, `CONTRIBUTING.md`, `CLAUDE.md`.
Expand Down
12 changes: 10 additions & 2 deletions docs/webhooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,13 @@ src/modules/webhooks/
├── services/
│ ├── webhook.service.ts ← CRUD operations, per-key webhook cap
│ ├── webhook-dispatch.service.ts ← Outbox: one delivery per subscribed webhook, in the caller's transaction (#184)
│ ├── webhook-delivery.service.ts ← Fair claim, attempts, replay, retention purge
│ ├── webhook-delivery.service.ts ← Fair claim, attempts, replay, retention purge: the public surface, re-exported from the files below (#387)
│ ├── webhook-delivery-schedule.service.ts ← Retry backoff (1m, 5m, 15m, 1h) and the 5-attempt cap
│ ├── webhook-delivery-claim.service.ts ← Fair claim across API keys (#258), attempt lease, signing secret decryption (#259)
│ ├── webhook-delivery-attempt.service.ts ← One HTTP attempt: SSRF re-validation, DNS pinning (#128), no redirects, no decompression, bounded read
│ ├── webhook-delivery-query.service.ts ← List, get and replay (replay budget, #258), scoped by API key
│ ├── webhook-delivery-outcome.service.ts ← Records an attempt: delivered, rescheduled or failed
│ ├── webhook-delivery-retention.service.ts ← Retention purge of finished rows (#258)
│ ├── webhook-delivery-worker.service.ts ← Poll loop and hourly cleanup
│ └── replay-limit.service.ts ← Replay budget per delivery (in memory)
├── dtos/
Expand Down Expand Up @@ -451,10 +457,12 @@ Single-webhook lookup scoped to an API key. Used by the deliveries endpoint to d
The outbox insert (#184): in the caller's transaction `tx`, one `INSERT ... SELECT` writes a `pending` `webhook_deliveries` row for every active webhook of an active key subscribed to `event`, limited by `scope` (`DispatchScope`): `{ apiKeyId }` for an email's own events (only that key's webhooks), or `{ adminOnly: true }` for `email.received` and the domain events (only admin keys' webhooks). Owner and event are matched in SQL. Returns the number of rows; zero is success. Throws `WebhookEnqueueError` (SQLSTATE only) when the insert fails. See [Transactional outbox](#transactional-outbox-184).

#### `signPayload(timestamp, body, secret): string`
HMAC-SHA256 over `<timestamp>.<body>`. Re-exported and called by the worker once per attempt.
HMAC-SHA256 over `<timestamp>.<body>`. Called once per attempt by `performHttpAttempt` (`webhook-delivery-attempt.service.ts`).

### webhook-delivery.service.ts (#30)

The service's public surface; the code sits in the `webhook-delivery-*.service.ts` files beside it, one per concern (#387, see the module layout above).

#### `claimDueDeliveries(n, now?): Promise<Array<...>>`
Atomically claims up to `n` due rows with `FOR UPDATE SKIP LOCKED`, shared round-robin across the API keys owning their webhooks (#258: a `LATERAL` per webhook takes its first `n` due rows, ranked within the owning key, lowest ranks first), and increments their `attempts` (each row's returned `attempts` is the count before this attempt). Inactive webhooks' rows are flipped to `failed` rather than delivered, and so are rows over `MAX_DELIVERY_ATTEMPTS` (attempts that never reached `recordAttempt` because the process died).

Expand Down
2 changes: 1 addition & 1 deletion src/config/retention.ts
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ export function buildIdempotencyConfig() {
* Webhook delivery queue (#30). Persisted retry loop config:
* tuned for the realistic consumer-outage profile rather than burst
* latency. The retry schedule itself (waits of 1m / 5m / 15m / 1h, so
* 5 attempts over about 80 minutes) lives in `webhook-delivery.service.ts`
* 5 attempts over about 80 minutes) lives in `webhook-delivery-schedule.service.ts`
* because it's a behavioural contract, not an operator knob.
*/
export function buildWebhookDeliveryConfig() {
Expand Down
Loading
Loading