Skip to content

Commit f400d5b

Browse files
authored
fix(transcript): load large tool images on demand instead of inline (#1118)
A session whose history holds image tool results (for example `read` on PNG files) never opened over a slow link (#1113): the 200-message page was 16.8 MB, 15.7 MB of it base64 in five tool results. It took 20 s at 6.5 Mbit/s, and a reverse proxy cut the transfer, so the UI showed "Failed to fetch". Every open, reload and history window paid for every image, shown or not. Transcript pages now carry large tool images by reference. The UI's native message list reads (createInstanceFetch) and history windows send the `x-codenomad-media: deferred` opt-in. For those reads only, the server replaces each tool image whose data URI exceeds 32 KiB with a `/api/workspaces/<id>/session-media/<session>/<message>/<part>/<index>/ <sha256>` path, in the instance proxy (forwardRuntimeRequest gains an optional whole-JSON transform for successful JSON responses) and in the history window route. toolImageSource resolves that path on the API base, so the existing lazy `<img>` loads only the images that are shown, as binary rather than base64. The new session-media route checks that the workspace owns the session, reads the native message, and refuses a session that moved during the read and messages at or after an undo boundary. It serves the bytes only when the image still matches the digest, as an immutable, nosniff, sandboxed image; anything else falls back to "Unable to display this image". What stays whole: - Single-message reads (`session.message.get`), so pruning revisions keep hashing native content. - Live events, user attachments, small images, non-image files and error responses. - Requests without the opt-in. The opt-in header is never forwarded to OpenCode. On the reported session, the page drops from 16.8 MB to about 1.1 MB. Validation: - session-media.test.ts: only large renderable tool images change, and only their URI. - routes/session-media.test.ts: exact bytes and headers; invalid references, digest and position mismatches, the undo boundary, ownership and a move during the read. - proxy.test.ts: the transform applies to successful JSON only and releases once. - instance-proxy.test.ts: an opted-in list read is deferred, the opt-in is not forwarded, and plain list or single-message reads stay whole. - session-navigation.test.ts: history windows. - sdk-manager.test.ts: only list reads opt in, keeping other headers. - tool-content.test.ts: path resolution, forged paths rejected; it is now listed in the PR build. - tool-images.test.ts: a deferred image decodes from the route in the real ToolCall renderer, and a missing one shows the fallback. The UI own-tsconfig, server and root typechecks pass, as do: - the server suite (2,467 passed; one index.test.ts process test timed out under load and passes alone, as on dev); - the UI unit step of the PR build (564 tests); - seven browser suites (278 tests).
1 parent 47e0279 commit f400d5b

20 files changed

Lines changed: 377 additions & 3 deletions

‎.github/workflows/pr-build.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -421,6 +421,7 @@ jobs:
421421
packages/ui/src/components/session/provider-usage-panel.test.ts
422422
packages/ui/src/lib/hooks/use-active-session-message-load.test.ts
423423
packages/ui/src/lib/sdk-manager.test.ts
424+
packages/ui/src/lib/tool-content.test.ts
424425
packages/ui/src/lib/background-read-queue.test.ts
425426
packages/ui/src/lib/prioritized-read.test.ts
426427
packages/ui/src/stores/instance-startup-priority.test.ts

‎dev-docs/TRANSCRIPT.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Message content visibility and presets, token lines, tool renderers, native sess
99
- Status Tokens are independent of message token visibility. A collapsed section uses the non-interactive `renderSummary` slot for one ellipsized session input/output/cost line; expanded content shows session metrics then descendant-inclusive totals as separate wrapping groups. Keep this layout in `styles/panels/status-tokens.css`.
1010
- Message token disclosures keep one row in both states: collapsed shows the input/output/cost summary chips on a single ellipsized line; expanded swaps the same row to all six chips beside the chevron, which wrap onto further lines when narrow instead of truncating (expansion exists to read every value). Keep the metric renderer local and chip/row styles in `styles/messaging/message-usage.css`.
1111
- Tool-result images retain native `state.content` and render through the shared `components/tool-call/output-images.tsx` surface, including MCP and specialized tools. Keep image bytes out of text projections (copy/search/speech); image styles live in `styles/messaging/tool-call/images.css`.
12+
- Transcript pages carry large tool images by reference: the UI's native message list reads (`lib/sdk-manager.ts`) and history windows (`fetchHistoryWindow`) send the `x-codenomad-media: deferred` opt-in (`server/src/opencode/session-media-contract.ts`), and the server replaces each tool image data URI over `INLINE_MEDIA_LIMIT` (32 KiB) with a `/api/workspaces/<id>/session-media/<session>/<message>/<part>/<index>/<sha256>` path (`opencode/session-media.ts`). `toolImageSource` resolves that path on the API base, so the lazy `<img>` loads only shown images; `routes/session-media.ts` serves the bytes of the owned session's native message, pinned by the digest and refused at or after an undo boundary, as an immutable sandboxed image. Single-message reads (`session.message.get`: pruning revisions, previews), live events, user attachments and requests without the opt-in stay whole, so revisions hash native content.
1213
- Code Mode reads native `metadata.toolCalls` in `tool-call/renderers/execute-data.ts`; keep call-list styling in `styles/messaging/tool-call/execute.css`, with native image/output ownership in the shared tool shell.
1314
- Native websearch text is parsed conservatively by `tool-call/renderers/websearch-data.ts`; unknown/oversized responses use the shared raw renderer. Recognized provider-consent Forms retain the generic schema/answer path, with layout in `styles/messaging/tool-call/websearch.css`.
1415
- Session lifecycle features use OpenCode V2's published signals: one `session.execution.{succeeded,failed,interrupted}` per busy period (`lib/native-execution-events.ts`) and `session.retry.scheduled` for provider backoff. The schema-declared `session.idle`/`session.status` are never published; never wire a feature only to them. A terminal event is a hint until `session.active` confirms settlement, which alone announces "A session finished", only for root sessions ending in success, failure or an inactivity stop. Transcript reducers retire at settlement, deferred while prompts stay parked in their inbox or a user shell or unsettled tool call still needs reconciling there. Token/cost lines (`components/message-usage.tsx`) appear once per agent message: the row of an assistant message's final text (the one carrying "delete tools and reasoning") renders, once that message has native usage, the sum (`sumMessageUsage`) of its own usage and of every message holding a tool or reasoning part that cleanup selects (`technicalCleanupParts`). Reuse that selection rather than a separate projection. The line renders inside the content row, so it never shifts `<Index>` items or splits a technical group (whose Delete needs two tools); tool-only steps get no line. A message whose text is followed by tools also counts in the next message's line, as its tools belong to that selection. Sums cover the loaded transcript window only, like cleanup. Capture real streams with `scripts/test-session-lifecycle-native.mjs`.

‎packages/server/src/opencode/compatibility/proxy.test.ts‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,29 @@ test("retiring the actual connection preserves complete and streamed native 401
3939
} finally { await service.shutdown(); await app.close(); await upstream.close() }
4040
}
4141
})
42+
43+
test("a transform rewrites only successful JSON responses and releases each request once", async () => {
44+
const upstream = Fastify()
45+
upstream.get("/api/json", async () => ({ data: [{ uri: "inline" }] }))
46+
upstream.get("/api/missing", async (_request, reply) => reply.code(404).send({ data: [{ uri: "inline" }] }))
47+
upstream.get("/api/text", async (_request, reply) => reply.type("text/plain").send("plain"))
48+
await upstream.listen({ host: "127.0.0.1", port: 0 })
49+
const origin = `http://127.0.0.1:${(upstream.server.address() as { port: number }).port}`
50+
const app = Fastify()
51+
let releases = 0, invalidations = 0
52+
app.get("/:path", (request, reply) => forwardRuntimeRequest({
53+
request, reply, url: `${origin}/api/${(request.params as { path: string }).path}`, body: undefined, headers: {}, fetch,
54+
invalidate: () => { invalidations++ }, release: () => { releases++ },
55+
transform: body => { for (const item of (body as { data: Array<{ uri: string }> }).data) item.uri = "deferred" },
56+
}))
57+
try {
58+
const json = await app.inject("/json")
59+
assert.equal(json.statusCode, 200)
60+
assert.match(String(json.headers["content-type"]), /^application\/json/)
61+
assert.deepEqual(json.json(), { data: [{ uri: "deferred" }] })
62+
assert.deepEqual((await app.inject("/missing")).json(), { data: [{ uri: "inline" }] }, "errors pass through")
63+
assert.equal((await app.inject("/text")).body, "plain")
64+
assert.equal(releases, 3)
65+
assert.equal(invalidations, 0)
66+
} finally { await app.close(); await upstream.close() }
67+
})

‎packages/server/src/opencode/compatibility/proxy.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ export async function forwardRuntimeRequest(args: {
1010
fetch: typeof fetch
1111
release?: () => void
1212
invalidate: () => void
13+
/** Rewrites a successful JSON response in place before it is relayed whole. */
14+
transform?: (body: unknown) => void
1315
}) {
1416
const controller = new AbortController()
1517
let released = false
@@ -47,6 +49,19 @@ export async function forwardRuntimeRequest(args: {
4749
if (value !== null) args.reply.header(name, value)
4850
}
4951
if (!response.body) { release(); return args.reply.send() }
52+
if (args.transform && response.ok && /^application\/json\b/i.test(response.headers.get("content-type") ?? "")) {
53+
let text: string
54+
try { text = await response.text() } catch {
55+
// Like an interrupted stream: the connection stays; only this response failed.
56+
release()
57+
return args.reply.code(502).send({ error: "OpenCode response was interrupted" })
58+
}
59+
release()
60+
let value: unknown
61+
try { value = JSON.parse(text) } catch { return args.reply.send(text) }
62+
args.transform(value)
63+
return args.reply.send(JSON.stringify(value))
64+
}
5065
const stream = Readable.fromWeb(response.body as import("node:stream/web").ReadableStream)
5166
stream.once("end", release).once("close", release).once("error", release)
5267
return args.reply.send(stream)
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
/**
2+
* Opt-in request header of transcript page reads (native message list, history
3+
* window): large tool images arrive as session-media paths instead of inline
4+
* base64, and the renderer loads each one when it is shown. Shared with the UI.
5+
*/
6+
export const DEFERRED_MEDIA_HEADER = "x-codenomad-media"
7+
export const DEFERRED_MEDIA_VALUE = "deferred"
8+
9+
/** `/api/workspaces/<id>/session-media/<session>/<message>/<part>/<index>/<sha256>` */
10+
export function isSessionMediaPath(value: string): boolean {
11+
return /^\/api\/workspaces\/[^/?#]+\/session-media\/[^/?#]+\/[^/?#]+\/\d{1,6}\/\d{1,6}\/[a-f0-9]{64}$/.test(value)
12+
}
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
import assert from "node:assert/strict"
2+
import { test } from "node:test"
3+
import { deferToolMedia, INLINE_MEDIA_LIMIT, mediaDigest, toolFile } from "./session-media"
4+
import { isSessionMediaPath } from "./session-media-contract"
5+
6+
const image = (bytes: number, mime = "image/png") => ({ type: "file", mime, name: "shot.png", uri: `data:${mime};base64,${"A".repeat(bytes)}` })
7+
8+
test("large tool images become digest-pinned session-media paths; everything else stays inline", () => {
9+
const large = image(INLINE_MEDIA_LIMIT), small = image(1024)
10+
const mismatched = { ...image(INLINE_MEDIA_LIMIT), mime: "image/jpeg" }
11+
const text = { type: "text", text: "A".repeat(INLINE_MEDIA_LIMIT * 2) }
12+
const pdf = image(INLINE_MEDIA_LIMIT, "application/pdf")
13+
const messages = [
14+
{ id: "msg_user", type: "user", text: "look", files: [image(INLINE_MEDIA_LIMIT)] },
15+
{ id: "msg_assistant", type: "assistant", content: [
16+
{ type: "text", text: "Reading" },
17+
{ type: "tool", id: "call", name: "read", state: { status: "completed", content: [text, structuredClone(large), small, mismatched, pdf] } },
18+
] },
19+
]
20+
const before = structuredClone(messages)
21+
deferToolMedia(messages, { workspaceId: "w/1", sessionId: "ses_1" })
22+
23+
const files = (messages[1] as any).content[1].state.content
24+
const path = files[1].uri as string
25+
assert.equal(path, `/api/workspaces/w%2F1/session-media/ses_1/msg_assistant/1/1/${mediaDigest(large.uri)}`)
26+
assert.ok(isSessionMediaPath(path))
27+
assert.deepEqual({ ...files[1], uri: large.uri }, large, "only the URI changes")
28+
assert.deepEqual([files[0], files[2], files[3], files[4]], [text, small, mismatched, pdf])
29+
assert.deepEqual(messages[0], before[0], "user attachments are not tool results")
30+
assert.equal(toolFile(before[1], 1, 1), (before[1] as any).content[1].state.content[1])
31+
assert.equal(toolFile(before[1], 0, 0), undefined)
32+
for (const value of [undefined, null, {}, [null, { type: "assistant" }, { type: "assistant", id: "m", content: [null, { type: "tool" }] }]]) {
33+
assert.doesNotThrow(() => deferToolMedia(value, { workspaceId: "w", sessionId: "s" }))
34+
}
35+
})
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
import { createHash } from "node:crypto"
2+
import { DEFERRED_MEDIA_HEADER, DEFERRED_MEDIA_VALUE } from "./session-media-contract"
3+
4+
export { DEFERRED_MEDIA_HEADER } from "./session-media-contract"
5+
/** Data URIs up to this length stay inline. */
6+
export const INLINE_MEDIA_LIMIT = 32 * 1024
7+
8+
type ToolImage = { type: "file"; mime: string; uri: string }
9+
10+
export function wantsDeferredMedia(headers: Record<string, string | string[] | undefined>): boolean {
11+
return headers[DEFERRED_MEDIA_HEADER] === DEFERRED_MEDIA_VALUE
12+
}
13+
14+
export function mediaDigest(uri: string): string {
15+
return createHash("sha256").update(uri).digest("hex")
16+
}
17+
18+
/** An image the renderer shows inline: a base64 data URI of its declared image type. */
19+
export function isRenderableToolImage(value: unknown): value is ToolImage {
20+
const file = value as Partial<ToolImage> | null
21+
if (file?.type !== "file" || typeof file.mime !== "string" || !/^image\/[\w.+-]+$/i.test(file.mime) || typeof file.uri !== "string") return false
22+
const header = `data:${file.mime.toLowerCase()};base64,`
23+
return file.uri.slice(0, header.length).toLowerCase() === header
24+
}
25+
26+
export function sessionMediaPath(target: { workspaceId: string; sessionId: string; messageId: string; part: number; index: number; digest: string }): string {
27+
const segment = encodeURIComponent
28+
return `/api/workspaces/${segment(target.workspaceId)}/session-media/${segment(target.sessionId)}/${segment(target.messageId)}/${target.part}/${target.index}/${target.digest}`
29+
}
30+
31+
/** A tool result file of a native assistant message, addressed by content positions. */
32+
export function toolFile(message: unknown, part: number, index: number): unknown {
33+
const native = message as { type?: unknown; content?: unknown } | null
34+
if (native?.type !== "assistant" || !Array.isArray(native.content)) return undefined
35+
const tool = native.content[part] as { type?: unknown; state?: { content?: unknown } } | undefined
36+
return tool?.type === "tool" && Array.isArray(tool.state?.content) ? tool.state.content[index] : undefined
37+
}
38+
39+
/**
40+
* Replaces, in place, tool images whose data URI exceeds INLINE_MEDIA_LIMIT in
41+
* native messages with their session-media path. The digest pins the exact bytes.
42+
*/
43+
export function deferToolMedia(messages: unknown, scope: { workspaceId: string; sessionId: string }): void {
44+
if (!Array.isArray(messages)) return
45+
for (const message of messages) {
46+
if (message?.type !== "assistant" || typeof message.id !== "string" || !Array.isArray(message.content)) continue
47+
message.content.forEach((tool: { type?: unknown; state?: { content?: unknown } } | null, part: number) => {
48+
if (tool?.type !== "tool" || !Array.isArray(tool.state?.content)) return
49+
tool.state.content.forEach((file: unknown, index: number) => {
50+
if (!isRenderableToolImage(file) || file.uri.length <= INLINE_MEDIA_LIMIT) return
51+
file.uri = sessionMediaPath({ ...scope, messageId: message.id, part, index, digest: mediaDigest(file.uri) })
52+
})
53+
})
54+
}
55+
}

‎packages/server/src/server/__tests__/instance-proxy.test.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,12 @@ async function harness(
6060
markDelayedUpstreamStarted()
6161
await new Promise<void>((resolve) => { releaseDelayedUpstream = resolve })
6262
}
63+
if (request.headers["x-test-message-list"] === "1") {
64+
const uri = `data:image/png;base64,${"A".repeat(64 * 1024)}`
65+
return { data: [{ id: "msg_1", type: "assistant", content: [{ type: "tool", id: "call", name: "read",
66+
state: { status: "completed", content: [{ type: "file", mime: "image/png", uri }] } }] }], cursor: {},
67+
forwarded: request.headers["x-codenomad-media"] ?? null }
68+
}
6369
reply.header("set-cookie", "upstream_session=secret; Path=/")
6470
reply.header("www-authenticate", 'Basic realm="OpenCode"')
6571
reply.header("proxy-authenticate", 'Basic realm="OpenCode proxy"')
@@ -307,6 +313,18 @@ describe("instance proxy location enforcement", () => {
307313
assert.equal(denied.statusCode, 403)
308314
})
309315

316+
it("defers large tool images only in opted-in transcript list reads, never forwarding the opt-in", async () => {
317+
const { app } = await harness()
318+
const read = async (path: string, optIn: boolean) => (await app.inject({ url: `/workspaces/workspace/instance/api/session/session/${path}`,
319+
headers: { "x-test-message-list": "1", ...(optIn ? { "x-codenomad-media": "deferred" } : {}) } })).json()
320+
const uri = (body: any) => body.data[0].content[0].state.content[0].uri as string
321+
const deferred = await read("message?limit=200&order=desc", true)
322+
assert.match(uri(deferred), /^\/api\/workspaces\/workspace\/session-media\/session\/msg_1\/0\/0\/[a-f0-9]{64}$/)
323+
assert.equal(deferred.forwarded, null)
324+
assert.match(uri(await read("message?limit=200", false)), /^data:image\/png;base64,/)
325+
assert.match(uri(await read("message/msg_1", true)), /^data:image\/png;base64,/, "a single message read stays whole")
326+
})
327+
310328
it("forwards native execution settlement only for a session owned by the workspace", async () => {
311329
const { app, requestCount, sessionGets } = await harness()
312330
const response = await app.inject({ method: "POST", url: "/workspaces/workspace/instance/api/experimental/session/session/wait" })

‎packages/server/src/server/http-server.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ import { registerYoloRoutes } from "./routes/yolo"
2828
import { registerPermissionReceiptRoutes } from "./routes/permission-receipts"
2929
import type { PermissionReceipts } from "../permissions/receipts"
3030
import { registerSessionPruningRoutes } from "./routes/session-pruning"
31+
import { registerSessionMediaRoutes } from "./routes/session-media"
3132
import { registerWorktreeRoutes } from "./routes/worktrees"
3233
import { registerPendingRequestRoutes } from "./routes/pending-requests"
3334
import { registerSpeechRoutes } from "./routes/speech"
@@ -74,6 +75,7 @@ import { buildPreviewRuntimeBridge, rewritePreviewImportMap, rewritePreviewJavaS
7475
import { forwardRuntimeRequest } from "../opencode/compatibility/proxy"
7576
import { requestAdmission } from "./request-admission"
7677
import { LOCATION_CONTEXT_HEADER, locationRequestOptions, readLocationContext } from "../opencode/compatibility/location"
78+
import { DEFERRED_MEDIA_HEADER, deferToolMedia, wantsDeferredMedia } from "../opencode/session-media"
7779
import { decodeSessionListScope, prepareLocationImport, readRequestLocations, type SessionListScope } from "../opencode/compatibility/proxy-locations"
7880
import type { RemoteProxySessionManager } from "./remote-proxy"
7981
import { createOpenCodeUpdateService } from "../opencode-update/service"
@@ -395,6 +397,7 @@ export function createHttpServer(deps: HttpServerDeps) {
395397
registerYoloRoutes(app, { yoloManager: deps.yoloManager })
396398
if (deps.permissionReceipts) registerPermissionReceiptRoutes(app, deps.permissionReceipts)
397399
registerSessionPruningRoutes(app, { workspaceManager: deps.workspaceManager, worktreeDeletionFence })
400+
registerSessionMediaRoutes(app, { workspaceManager: deps.workspaceManager })
398401
registerInstanceProxyRoutes(app, { workspaceManager: deps.workspaceManager, logger: proxyLogger, worktreeDeletionFence, accounts, permissionReceipts: deps.permissionReceipts,
399402
humanAnswers: { auth: deps.authManager, manager: deps.workspaceManager, settings: deps.settings, bridgeToken: deps.automationBridgeToken,
400403
remoteDevices: deps.remoteControlManager } })
@@ -1169,7 +1172,13 @@ async function proxyWorkspaceAdmission(args: InstanceProxyRequestArgs, admission
11691172
if (globalFormLocation) headers["x-opencode-directory"] = encodeURIComponent(translatedDirectories.get(globalFormLocation.directory)!)
11701173
const runtimeFetch = connection?.fetch ?? await wait(workspaceManager.getSharedServiceFetch!())
11711174
signal.throwIfAborted()
1175+
// A transcript page read by the UI: large tool images load on demand.
1176+
const deferMedia = request.method === "GET" && /^\/api\/session\/[^/]+\/message\/?$/.test(pathname)
1177+
&& wantsDeferredMedia(request.headers)
11721178
return await forwardRuntimeRequest({
1179+
transform: deferMedia && sessionId
1180+
? body => deferToolMedia((body as { data?: unknown } | null)?.data, { workspaceId, sessionId })
1181+
: undefined,
11731182
request, reply, url: targetUrl.toString(), body, headers,
11741183
fetch: confirmPermission || confirmDeletion ? async (input, init) => {
11751184
const response = await runtimeFetch(input, init)
@@ -1286,6 +1295,7 @@ function sanitizeInstanceProxyRequestHeaders(
12861295
"x-forwarded-for", "x-forwarded-host", "x-forwarded-port", "x-forwarded-proto",
12871296
LOCATION_CONTEXT_HEADER,
12881297
HUMAN_ANSWER_HEADER,
1298+
DEFERRED_MEDIA_HEADER,
12891299
])
12901300
const connection = headers.connection
12911301
for (const name of (Array.isArray(connection) ? connection.join(",") : connection ?? "").split(",")) blocked.add(name.trim().toLowerCase())

0 commit comments

Comments
 (0)