Skip to content

fix(inbound): keep email.received events under the EventBridge size limit #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stewartjarod merged 1 commit into from
Oct 21, 2026
Merged

fix(inbound): keep email.received events under the EventBridge size limit #197

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 101 additions & 1 deletion apps/website/src/app/docs/infrastructure/events/page-content.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1562,7 +1562,7 @@ export default function InfrastructureEventsPageContent() {
Email subject line
</td>
</tr>
<tr>
<tr className="border-b">
<td className="p-3">
<code className="rounded bg-muted px-1 py-0.5">
detail.receivedAt
Expand All @@ -1572,6 +1572,41 @@ export default function InfrastructureEventsPageContent() {
When the inbound email was received
</td>
</tr>
<tr className="border-b">
<td className="p-3">
<code className="rounded bg-muted px-1 py-0.5">
detail.rawS3Key
</code>
</td>
<td className="p-3 text-muted-foreground">
S3 key of the raw MIME message in your inbound
bucket
</td>
</tr>
<tr className="border-b">
<td className="p-3">
<code className="rounded bg-muted px-1 py-0.5">
detail.truncated
</code>
</td>
<td className="p-3 text-muted-foreground">
Present and <code>true</code> only when large fields
were dropped to fit the EventBridge size limit;{" "}
<code>detail.truncatedFields</code> lists them
</td>
</tr>
<tr>
<td className="p-3">
<code className="rounded bg-muted px-1 py-0.5">
detail.parsedS3Key
</code>
</td>
<td className="p-3 text-muted-foreground">
Set on truncated events: S3 key of the full parsed
email JSON, in the same bucket as{" "}
<code>rawS3Key</code>
</td>
</tr>
</tbody>
</table>
</div>
Expand All @@ -1581,6 +1616,71 @@ export default function InfrastructureEventsPageContent() {
code={inboundPayload}
filename="inbound-event.json"
/>
<p className="text-muted-foreground text-sm">
The full parsed email is always stored in your bucket at{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
parsed/{"{emailId}"}.json
</code>
. If it would exceed the EventBridge PutEvents size limit (1 MB,
or 256 KB in Regions without the 1 MB increase), the event is
trimmed: the largest of{" "}
<code className="rounded bg-muted px-1.5 py-0.5">html</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">text</code> and{" "}
<code className="rounded bg-muted px-1.5 py-0.5">headers</code>{" "}
is dropped first, then the others, then{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
attachments
</code>
, until it fits, and the detail gains{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
truncated: true
</code>
,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
truncatedFields
</code>{" "}
and{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
parsedS3Key
</code>
. When{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
truncated
</code>{" "}
is set, read the full email from{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
parsedS3Key
</code>{" "}
(or the raw MIME from{" "}
<code className="rounded bg-muted px-1.5 py-0.5">rawS3Key</code>
). If that is still too large,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">to</code>/
<code className="rounded bg-muted px-1.5 py-0.5">cc</code> are
replaced by{" "}
<code className="rounded bg-muted px-1.5 py-0.5">toCount</code>/
<code className="rounded bg-muted px-1.5 py-0.5">ccCount</code>{" "}
and the subject is cut to 1,000 characters; as a last resort the
detail holds only{" "}
<code className="rounded bg-muted px-1.5 py-0.5">emailId</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">rawS3Key</code>
,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
parsedS3Key
</code>
,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
receivedAt
</code>
,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
replyToken
</code>{" "}
and{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
autoReply
</code>
.
</p>
<p className="text-muted-foreground text-sm">
When replying to a signed message,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
Expand Down
30 changes: 30 additions & 0 deletions apps/website/src/app/docs/quickstart/email/inbound/page-content.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -687,6 +687,36 @@ TTL: 300`,
.
</p>

<p className="mb-4 text-muted-foreground text-sm">
Very large emails are trimmed to fit the EventBridge size limit: the
event then has{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
truncated: true
</code>
, drops <code className="rounded bg-muted px-1.5 py-0.5">html</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">text</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">headers</code> or{" "}
<code className="rounded bg-muted px-1.5 py-0.5">attachments</code>{" "}
(listed in{" "}
<code className="rounded bg-muted px-1.5 py-0.5">
truncatedFields
</code>
), and gives the full parsed email's S3 key in{" "}
<code className="rounded bg-muted px-1.5 py-0.5">parsedS3Key</code>.
Sender, recipients, subject and verdicts are kept unless they alone
are too large; then{" "}
<code className="rounded bg-muted px-1.5 py-0.5">to</code>/
<code className="rounded bg-muted px-1.5 py-0.5">cc</code> are
replaced by counts and the subject is cut, and as a last resort only{" "}
<code className="rounded bg-muted px-1.5 py-0.5">emailId</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">rawS3Key</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">parsedS3Key</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">receivedAt</code>,{" "}
<code className="rounded bg-muted px-1.5 py-0.5">replyToken</code> and{" "}
<code className="rounded bg-muted px-1.5 py-0.5">autoReply</code> are
sent.
</p>

<p className="mb-2 font-medium text-sm">Example Lambda Handler</p>
<CodeBlock
className="h-auto"
Expand Down
11 changes: 9 additions & 2 deletions packages/core/CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,18 @@ These resolve to the pre-bundled Lambda code location for Pulumi/CDK to deploy.
- Uses `mailparser` to parse raw MIME
- Extracts headers, body (max 200KB HTML), attachments
- Uploads attachments to S3 with sanitized filenames
- Publishes EventBridge event `email.received`
- Stores the full parsed email at `parsed/{emailId}.json`
- Publishes EventBridge event `email.received` on the default bus. If the entry
would exceed the PutEvents limit (1 MB; 256 KB in the Regions listed in
`LEGACY_LIMIT_REGIONS` and GovCloud), `buildEventDetail` drops `html`/`text`/`headers`
(largest first), then `attachments`, then `to`/`cc` (and cuts the subject),
and as a last resort sends a bounded envelope; it adds `truncated: true`,
`truncatedFields` and `parsedS3Key`. Small mail is sent unchanged.
- Throws when PutEvents returns `FailedEntryCount > 0`
- Extracts SES spam/virus verdicts from headers
- **Re-throws errors** (unlike other processors) to trigger retry/DLQ

**Env vars**: `PARSED_BUCKET`, `EVENT_BUS_NAME`, `WRAPS_ACCOUNT_ID`, `AWS_REGION`
**Env vars**: `BUCKET_NAME`, `INBOUND_EVENT_SOURCE` (default `wraps.inbound`), `REPLY_SECRET_PARAMETER_PREFIX` (optional), `AWS_REGION`

## Lambda Loading Priority (in CLI)

Expand Down
Loading
Loading