[codex] Support raw image offload in v1 train client

[eligotts]

Design update — inline/offload image storage

This PR now follows the prime-rl multimodal image storage policy:

• offload: current behavior, rewrite base64 data images to file:// run assets and require file-backed image URLs.
• inline: keep data:image/...;base64,... URLs in the message payload and validate them without rewriting.

TrainClient now calls the policy-aware image preparation helper, so prime-rl can be the single source of truth via environment/config propagation.

Validation after latest push: uv run pytest tests/v1/test_train_client_multimodal.py -q passed (5 passed). Commit/push hooks also passed (ruff check, ruff format, generated AGENTS/CLAUDE check, ty).

Design update — dropped the None/cache-only image path

This PR and its companions (prime-rl #2836 / verifiers #1746 / renderers #89) no longer use the "send None for already-cached images" mechanism. Every image carries its raw descriptor ref at every slot (current and prior turns); /inference/v1/generate rematerializes each ref from disk every request.

Why: the None path coupled correctness to deployment (LRU cache present, single replica / DP-affinity, no eviction) and surfaced a miss as a hard vLLM EngineDeadError (qwen3-vl mrope dereferences a None image_grid_thw) that the retry net couldn't catch across the engine→API IPC. Dropping it is deployment-agnostic (a miss is impossible) and non-hacky. vLLM's mm_hash encoder cache still skips the expensive GPU re-encode for free — we only forgo the cheap IPC/CPU-reprocess dedup.

Validated: color-codeword (Qwen3-VL-4B) under DP=2, no affinity / no cache reliance: 0 crashes, 0 data=None, multi-turn accumulation correct, reward ~0.84. Also confirmed under TP.

This repo: with every image carrying its ref, no cache miss can occur — removed the retry subsystem (_generate_with_image_ref_retry, _has_descriptor_only_images, _retryable_mm_error_type, _json_error_type, _RETRYABLE_MM_ERROR_TYPES). Rollouts call renderers.client.generate directly. Obsolete retry tests removed.

---

Original description

Summary

• tighten v1 multimodal graph serialization around strict raw descriptor sidecars
• reject processed multimodal payload keys recursively, including nested pixel_values, image_embeds, and image_features
• update v1 multimodal tests to use strict prime_raw_mm_item envelopes instead of descriptor-only Qwen payloads
• keep raw image offload and retry behavior aligned with the companion Renderers and Prime-RL PRs

Companion PRs

• Prime-RL: Support v1 raw multimodal image offload prime-rl#2836
• Renderers: [codex] Support raw image refs for multimodal rendering renderers#89

Notes

• Draft/WIP: this depends on the renderer generic raw multimodal ref contract in the companion PR.
• v1 multimodal sidecars intentionally carry raw descriptors only, not processed image tensors or image-processor payloads.
• Prime/vLLM materialization happens from raw image refs rather than Verifiers-held processor outputs.

Validation

• Commit hooks: ruff check, ruff format, generated AGENTS/CLAUDE check passed.
• Push hook: ty (ci parity) passed.
• End-to-end hosted-style smoke through Prime-RL with /home/ubuntu/verifiers, /home/ubuntu/renderers, and /home/ubuntu/prime-rl-v1-raw-mm-offload completed inference, env rollouts, train batch creation, trainer step 0, and decoded strict trainer-bound raw image refs.

[!NOTE]

Add raw image offload support to v1 train client

• Adds prepare_images_inplace in verifiers/utils/multimodal.py that recursively walks message content, offloads image URLs to run assets, and rewrites them as file:// URIs; raises RuntimeError on non-file URLs that cannot be offloaded.
• TrainClient overrides prepare_request_body to call prepare_images_inplace on chat requests, and RendererClient.to_native_prompt runs the same preprocessing before converting to renderer messages.
• MessageNode in verifiers/v1/graph.py now validates that multi_modal_data carries raw-image descriptors (not processed pixel tensors), and gains finish_reason and usage fields.
• Multimodal bridging across turns is now enabled for TrainClient by passing previous_multi_modal_data to the renderer; previously such turns could not bridge.
• Risk: any image URL that is not a file:// URI and cannot be offloaded will cause the request to be rejected with a RuntimeError.

^{Macroscope summarized 88c759b.}

Update: ingress hardening (2c2824ae)

• prepare_images_inplace now covers the full renderer part treaty: nested image_url dicts, direct-string image_url, direct-string image parts, and typed pydantic parts. Non-string sources raise with the part shape named; renderer-side raw mode hard-requires file:// (no second offload layer).
• Interception server labels prepare_messages failures as InterceptionError instead of misattributing them to the user simulator.
• New shape-coverage test (skips until the renderers pin ships mm_store; passes against the sibling renderers checkout). Suite: 839 passed with pre-existing test_envs/test_opencode_rlm_env failures reproduced on the base branch.

Update: merged main (2b1627d03)

Reconciled with verifiers main (111 commits ahead at the merge-base — mostly unrelated v1 harness/multi-agent, taskset/environment, and trace/data-model churn; none of it touched this branch's actual offload files, verifiers/utils/multimodal.py / clients/renderer_client.py / types.py).

• interception/server.py: main's d21100bea ("resume replaces mid-request user injection") independently moved the user-simulator loop out of the interception server entirely, into Harness.launch/resume, and added graph-atomicity retry coalescing to handle_request/_stream. Took main's structure as authoritative; dropped this branch's now-obsolete session.user/session.opening mid-request injection (and its prepare_messages call sites) since injected/resumed messages now re-enter as ordinary request bodies, already covered by prepare_request_body (kept, rewired to the top of the new single-shot handle_request).
• graph.py: kept finish_reason/usage/multi_modal_data/previous_multi_modal_data() (this branch) alongside main's SkipJsonSchema wrapping convention and commit()'s new (tools) -> assistant_node_id signature. Caught and fixed an auto-merge that silently dropped the FinishReason/Usage imports — surfaced as a pydantic "Trace not fully defined" failure at Trace construction, not a textual conflict.
• trace.py: kept this branch's more accurate multi_modal_data docstring; took main's tuple-based bridge-mutation assertion in the test suite (strictly better — reuses already-computed prior_mm/prior_counts instead of recomputing).
• verifiers/v1/ARCHITECTURE.md: main deleted this file (docs moved to a much shorter docs/v1/architecture.md that doesn't cover this depth) — accepted the deletion; the one load-bearing fact (why multi_modal_data is JSON-excluded) is now a docstring on _NODE_DUMP_EXCLUDE instead of a dedicated doc.

Validation: full suite passing with a local renderers checkout override (uv run --with-editable ../renderers pytest tests/, minus PRIME_API_KEY-gated e2e/env tests) — includes test_prepare_images_inplace_offloads_every_image_part_shape, previously skipped pending a renderers pin with mm_store.

Update: dropped the orphaned prepare_messages hook (d9e79d6c)

Main's harness-segment-resume refactor removed the interception server's mid-request user-simulator injection — the only call sites of this PR's prepare_messages hook. Resumed user turns now re-enter the server as ordinary request bodies, already covered by prepare_request_body at request ingress, so the hook (base Client + TrainClient override) is deleted rather than carried as dead code.

Update: synced with main (0e4166608)

Merged verifiers main (35 commits): the v1 structure cleanup (#2146) retiring StrictBaseModel and renaming _NODE_DUMP_EXCLUDE → EXCLUDE_FIELDS, the ruff 0.16 / ty tooling bump (#2147, #2148), runtime-on-agent with the agent config stamped on the trace (#2106), Reward score/weight records (#2119), MCP tools for Codex (#2140).

Resolutions:

• trace.py: main's restructure subsumes this branch's block wholesale — EXCLUDE_FIELDS already carries multi_modal_data, and TRACE_VERSION = 1 is main's deliberate reset (this branch never touched it). Took main.
• graph.py: kept the raw-mm sidecar validators and previous_multi_modal_data; adopted main's plain BaseModel now that StrictBaseModel is gone.
• clients/train.py: kept the is_multimodal import — still used by the bridge path that threads previous_multi_modal_data.

Ruff 0.16 flagged two spots this branch added that main's own cleanup pass never saw: a constant getattr in the ingress offload walker, and the blind except at the prepare_request_body boundary (annotated # noqa: BLE001, matching main's convention at rollout boundaries).

Full suite with the sibling renderers checkout (minus PRIME_API_KEY-gated e2e/envs): all passing.

---

Note

Medium Risk
Changes the multimodal training contract (mandatory file offload, stricter sidecar validation) and multi-turn bridge behavior; failures surface at request prep rather than deep in inference.

Overview
Multimodal training ingress now rewrites embedded image payloads (base64 data: URLs and the various image_url / image part shapes) into content-addressed file:// run assets via renderers.mm_store, and rejects any image that is not file-backed after preparation. The walker runs on wire dicts and typed message models; v0 RendererClient.to_native_prompt and v1 TrainClient.prepare_request_body invoke it (the interception server calls prepare_request_body up front and surfaces prep failures as InterceptionError).

v1 trace/graph multimodal shifts from processed tensors to raw image descriptor sidecars (raw_image_uri required; pixel_values / image_embeds / image_features rejected). Per-node attribution now keeps placeholder ranges alongside hashes/items; PendingTurn.previous_multi_modal_data() feeds multimodal bridge turns, and the old “skip bridge when multimodal” guard is removed. Branch multi_modal_data merges placeholders; legacy v0→v1 trace conversion keeps live cumulative multi_modal_data from trajectory state.

A regression test covers all supported image part shapes and HTTPS rejection; uv.lock picks up editable package metadata blocks.

^{Reviewed by Cursor Bugbot for commit 88c759b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces new multimodal image offload functionality with significant changes to training pipelines. There are unresolved review comments including a high-severity concern about image data potentially being silently dropped, plus backwards compatibility and validation issues that warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

🟡 Medium v1/graph.py:76

_validate_raw_mm_item now unconditionally rejects processed multimodal payloads containing keys like pixel_values, and deserialize_multi_modal_data runs it on every multi_modal_data field during deserialization. Loading a previously persisted multimodal v1 trace whose sidecars contain pixel_values now raises TypeError instead of round-tripping, breaking backwards compatibility for existing saved rollouts. Consider allowing processed payloads through on the deserialization path (e.g. by skipping the processed-key check in the validator's before path) so old traces can still be loaded.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/graph.py around line 76:

`_validate_raw_mm_item` now unconditionally rejects processed multimodal payloads containing keys like `pixel_values`, and `deserialize_multi_modal_data` runs it on every `multi_modal_data` field during deserialization. Loading a previously persisted multimodal v1 trace whose sidecars contain `pixel_values` now raises `TypeError` instead of round-tripping, breaking backwards compatibility for existing saved rollouts. Consider allowing processed payloads through on the deserialization path (e.g. by skipping the processed-key check in the validator's `before` path) so old traces can still be loaded.

[macroscopeapp]

🟡 Medium utils/multimodal.py:64

prepare_images_inplace skips validation when an image_url part has a missing or None image_url field: lines 65-67 only call _prepare_image_source when source is not None, so the malformed part passes through unchecked. Downstream, ChatDialect.parse_request normalizes it to ImageUrlSource(url=""), forwarding a request with an empty image URL instead of rejecting it. Consider calling _require_file_image_url(value) (or otherwise validating) when source is None so malformed parts are rejected.

Suggested change

	if value.get("type") == "image_url":
	source = value.get("image_url")
	if source is not None:
	_prepare_image_source(source, image_dir=image_dir)
	if value.get("type") == "image_url":
	source = value.get("image_url")
	if source is not None:
	_prepare_image_source(source, image_dir=image_dir)
	else:
	_require_file_image_url(value)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/utils/multimodal.py around lines 64-67:

`prepare_images_inplace` skips validation when an `image_url` part has a missing or `None` `image_url` field: lines 65-67 only call `_prepare_image_source` when `source is not None`, so the malformed part passes through unchecked. Downstream, `ChatDialect.parse_request` normalizes it to `ImageUrlSource(url="")`, forwarding a request with an empty image URL instead of rejecting it. Consider calling `_require_file_image_url(value)` (or otherwise validating) when `source` is `None` so malformed parts are rejected.

[macroscopeapp]

🟡 Medium utils/multimodal.py:52

_prepare_image_part calls _offload_image_url(url, image_dir) before checking whether url is already a file:// path. When a prompt already contains local file:// image URLs and the installed renderer build lacks offload_image_to_run_assets, _offload_image_url raises RuntimeError even though no offload was needed, causing valid pre-offloaded multimodal prompts to fail. Consider returning early when the source is already a file:// URL before invoking _offload_image_url.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/utils/multimodal.py around line 52:

`_prepare_image_part` calls `_offload_image_url(url, image_dir)` before checking whether `url` is already a `file://` path. When a prompt already contains local `file://` image URLs and the installed renderer build lacks `offload_image_to_run_assets`, `_offload_image_url` raises `RuntimeError` even though no offload was needed, causing valid pre-offloaded multimodal prompts to fail. Consider returning early when the source is already a `file://` URL before invoking `_offload_image_url`.

[macroscopeapp]

🟠 High utils/multimodal.py:27

_part_image_field returns "image" unchanged for parts with type == "image", so _prepare_image_part offloads the source but leaves the part in the non-canonical {"type": "image", "image": ...} shape. Downstream v1 chat parsing only preserves image_url parts, so the image is silently dropped from the traced/training prompt even after prepare_images_inplace runs. Consider normalizing image parts to image_url (or mapping image to the image_url field during offload) so downstream parsers retain them.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/utils/multimodal.py around line 27:

`_part_image_field` returns `"image"` unchanged for parts with `type == "image"`, so `_prepare_image_part` offloads the source but leaves the part in the non-canonical `{"type": "image", "image": ...}` shape. Downstream v1 chat parsing only preserves `image_url` parts, so the image is silently dropped from the traced/training prompt even after `prepare_images_inplace` runs. Consider normalizing `image` parts to `image_url` (or mapping `image` to the `image_url` field during offload) so downstream parsers retain them.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2b1627d. Configure here.}