Skip to content
This repository was archived by the owner on Nov 8, 2026. It is now read-only.

fix(memory): measure and trim the session log without reading it all - #3504

Open
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/o3vxu6xbtx88
Open

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/o3vxu6xbtx88

Conversation

@polylane

@polylane polylane Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Session logs that grow past the Durable Object memory limit were reset instead of trimmed. The byte policy measured a log by loading every stored turn into memory on each write, and its 200 MiB budget is larger than the 128 MB limit, so a large review transcript could never be measured or trimmed. Writes now measure and trim a log without loading all of it.

flowchart TD
  A["run writes a transcript turn"] --> B["SessionLogDO /runs/session/write"]
  B --> C["enforceBytePolicy()"]
  C --> D["totalBytes(): SELECT every row's json (observed)"]
  D --> E["isolate holds the whole log's JSON"]
  E --> F{"log size vs 128 MB isolate limit, budget 200 MiB"}
  F -->|over| G["isolate reset mid-write, in-flight turn lost"]
  C -.-> H["fix: SUM(length(CAST(json AS BLOB))) in SQLite"]
  C -.-> I["fix: page candidate, orphan and fence scans, 256 rows"]
Loading
Detection evidence

Affected: acc_017c57bfc0010p0u2hmnjl5u

Why this fix

The per-object metric showed one session log at 144.17 MiB against the 128 MB isolate limit, with a second at 123.31 MiB, and objects keep climbing toward it. Every turn write measured the log by selecting all of its stored JSON into the isolate, and the trim loop re-read it for each candidate, so a log larger than the isolate reset the object instead of shrinking. Summing bytes inside SQLite and reading rows in bounded pages removes that read. The 200 MiB budget is unchanged and intentional; a search over a very large transcript still reads the whole log and is tracked separately.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:21). Recorded for commit b18df597f57d74678c6862265d457c98b5059a56.

Out of scope
  • SessionLogDO.search() still reads the whole log when it holds internal transcripts; paging it is tracked on the issue timeline.

Production impact: Session log writes trim the same way and read less memory

I checked the switchboard-memory worker and its SessionLogDO namespace on coreplane-infra, which served 72,381 requests and 5 errors in the last 48 hours, with top objects at 123.31, 60.34 and 50.64 MiB. The changed methods are private to SessionLogDO; the byte-policy, trimming and attachment tests pass unchanged, and the SQL aggregate returns the same UTF-8 byte total the JS sum returned.

Checked and not affected: switchboard-memory, bcb98cc3b42b4c79b9896fd28579efdb.

Observability

  • Existing per-object metric durableObjectsPeriodicGroups.max.memoryUsageBytes on namespace switchboard-memory_SessionLogDO shows whether a log still approaches the 128 MB limit.
  • Existing POST /runs/session/write response carries bytes from enforceBytePolicy(), so the measured total after each write is visible per request.
  • Existing worker logs record a reset as an isolate memory-limit error on the caller if one still happens.

Polylane assessed this pull request at b18df59.

Causal chain
  • Signal (metric): durableObjectsPeriodicGroups.max.memoryUsageBytes for namespace switchboard-memory_SessionLogDO
  • Surfacing site: switchboard-memory_SessionLogDO (bcb98cc3b42b4c79b9896fd28579efdb) at coreplanelabs/switchboard:deploy/cloudflare-memory/worker.ts#SessionLogDO.totalBytes
  • Mechanism: Each POST /runs/session/write calls enforceBytePolicy(), whose totalBytes() selected every row's json into the isolate before trimming; with the budget at 200 MiB and the isolate limit at 128 MB, a log that grew past the limit was read whole and the object reset instead of trimming. The attachment-fence check in the same transaction read the whole turns table as well.
  • Producer: SessionLogDO, at coreplanelabs/switchboard:deploy/cloudflare-memory/worker.ts#SessionLogDO.enforceBytePolicy
  • Blast radius: 2 other resource(s), unknown other tenant(s); data at risk: The in-flight transcript turn of the thread whose SessionLogDO resets mid-write.
  • Producer evidence:
    • Per-object metric: object 704bd249… (slack:C0C5002RWJY:…:review:@context-v1) peaked at 151,175,639 bytes (144.17 MiB) in the 7 days to 2026-10-28; next objects 123.31 MiB and 96.88 MiB.
    • Code: deploy/cloudflare-memory/worker.ts#SessionLogDO.totalBytes ran SELECT idx,part,json FROM turns and summed utf8ByteLength in JS; enforceBytePolicy calls it on every write and re-measures each trim pass.
    • Code: the write transaction's attachment-fence check ran SELECT idx, json FROM turns (now SessionLogDO.pinnedRowReferences).
    • Config: DEFAULT_SESSION_LOG_MAX_BYTES (src/core/runRecord.ts) is 200 MiB, above the 128 MB Durable Object isolate limit, so the policy permits a log larger than the isolate.
    • Telemetry: the namespace served 72,381 requests and 5 errors in 48 hours; its top objects reached 123.31, 60.34 and 50.64 MiB, so logs approach the limit in normal traffic.
1 file changed (+113/-52)
  • deploy/cloudflare-memory/worker.ts: modified, +113/-52

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment.

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added the polylane label Oct 28, 2026
@polylane
polylane Bot requested a review from justinhelmer as a code owner October 28, 2026 03:06
@polylane polylane Bot added the severity:medium Polylane autofix severity: medium label Oct 28, 2026
@polylane
polylane Bot requested a review from boristane October 28, 2026 03:06
@polylane

polylane Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

This pull request is waiting on a decision: merge it if the change is still wanted, or close it if the fix is no longer needed. Either one settles it.

It was opened on 2026-10-28 and has had no new activity for a week. If nothing happens on it within a week, Polylane closes it and keeps the branch, so reopening it brings the change back exactly as it stands.

@justinhelmer and @boristane, the change touches what you own, so the review is with you.

This branch has not been deployed

No deployments
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant