Skip to content

Capture a macOS harness window offscreen, without Screen Recording (#506) - #516

Merged
yeroo merged 1 commit into
mainfrom
issue-506-macos-offscreen-capture
Sep 28, 2026
Merged

yeroo merged 1 commit into
mainfrom
issue-506-macos-offscreen-capture

Conversation

@yeroo

@yeroo yeroo commented Sep 28, 2026

Copy link
Copy Markdown
Owner

Part of #506: pixel capture on macOS. With this, shot, window and every pixel assertion work for a macOS harness instance, without Screen Recording permission and without the window ever being on screen. Windows is untouched.

Why the app has to take the picture

On Windows the harness photographs the window from outside with PrintWindow. On macOS it cannot. Since #511 a harness window is never on screen, and reading another process's pixels needs Screen Recording, which a test must not ask for.

So on macOS the app takes the picture itself. The new capture verb renders the last drawn frame to an offscreen texture with gpui's Window::render_to_image, writes raw RGBA to <sandbox>/suite/capture/last.rgba, and replies:

{"path": "...", "width": 1180, "height": 800, "scale": 1, "frame": 8,
 "content_origin": {"x": 370, "y": 140}}

uiharness reads the file, and from then on it is an ordinary Capture. content_origin is computed by the same screen_rect that rect answers with, so the existing crop (a region's rect minus the capture's origin) lands on the right pixels unchanged, and every border probe runs exactly as on Windows. A unit test pins that invariant on fractional origins, a Retina scale, and a display left of the main one. The pixels travel beside the control channel rather than through it, because a full window is megabytes of RGBA.

Driver::capture is the single seam: macOS calls the verb, and everything else still calls capture_pid.

The harness-capture feature

render_to_image only exists behind gpui's test-support, so it needs a build made for it:

cargo build --release --manifest-path suite/Cargo.toml \
  --features harness-capture --target-dir suite/target/capture
cargo run --release -p uiharness -- run uiharness/cases/sheet-selection.uit \
  --suite suite/target/capture/release/suite

Two details are load-bearing. Both were checked against the pinned gpui (8276687), not GitHub:

  • It forwards to gpui_platform/test-support, not gpui/test-support. The macOS implementation is gated on gpui_macos's own feature. Enabling only gpui's builds cleanly and then answers every capture with "render_to_image not implemented for this platform".
  • It is a compile_error! off macOS. test-support also makes gpui draw every dirty window as it flushes effects, without presenting. On Windows that would let a frame wait be satisfied by a frame that never reached the screen, and PrintWindow would photograph the one before it.

A default build answers capture with an error naming the feature, not a blank picture:

  ERROR shot cell:A1:C5
        this build cannot capture offscreen: rebuild the suite with `--features harness-capture` (macOS only) to take pictures of a harness window

The driver also refuses a blank render, a pixel file whose size does not match the stated dimensions, and a reply missing a field. A blank image reaching a probe would read "no border" and pass.

⚠️ Fidelity caveat

  • A macOS pixel case tests a test-support build. Its draw scheduling differs from the shipped app's, and leak detection is on. The scene is the same, but it is not the binary users run. Never ship this feature.
  • It renders gpui's scene, not what a compositor showed. Anything the system draws outside gpui is absent, most visibly the native window buttons, so window is content only where PrintWindow includes the frame.

Proof: macOS 27.0, M1, through the real runner

The unmodified uiharness/cases/sheet-selection.uit, a case that passes on Windows today. Thresholds tuned on Windows were left untouched:

case: drag-to-select does not fill — ok
        195x120 via offscreen render
case: a swept selection is solid, not dashed — ok
        every edge solid in teal (#2aa79b) (195x120 via offscreen render)
case: a pointed range dashes — ok
        every edge dashed in teal (#2aa79b) (130x96 via offscreen render)
case: a formula reference is washed, not dashed — ok
        every edge solid in #2f6fdb (130x96 via offscreen render)
case: one selection at a time — ok
5 cases — 5 passed, 0 failed

All three border assertions pass. During the run the window server reported the window as onscreen=nil, and the frontmost app never changed (lsappinfo, no permission prompt). The instance exits 0 with an empty stderr, so leak detection raised nothing at quit.

The whole window, from a window that was never on screen:

whole window, offscreen

The crops the assertions read:

solid teal, A1:C5 dashed teal, A1:B4 solid #2f6fdb, A1:B4
solid dashed washed

The images live on the evidence/issue-506-macos-capture branch, which holds only these PNGs and is not for merging. Delete it after merge.

Cost of the feature

Default harness-capture
Binary 21,702,768 bytes 22,137,088 bytes (+2%)
Distinct dependencies 439 451
Clean build, own target dir n/a 252 s (git dependencies already fetched)

The twelve extra crates are proptest and its helpers. Cargo.lock only gains packages: four small crates get a second version alongside the first, and the gpui pin is unchanged. No fork, patch or re-pin. CI is untouched.

Checks

cargo fmt --check                                       # suite + uiharness: clean
cargo clippy --all-targets -- -D warnings               # suite default: 0
cargo clippy --all-targets --features harness-capture -- -D warnings   # suite: 0
cargo clippy -p uiharness --all-targets -- -D warnings  # 0
cargo test --manifest-path suite/Cargo.toml             # 486 passed (both configurations)
cargo test -p uiharness                                 # 121 + integration tests passed

New tests: five for the driver's reply parsing and refusals, and three for the app side (capture path inside the sandbox, the crop invariant, and the refusal message).

For review on Windows

  • cargo build --manifest-path suite/Cargo.toml --features harness-capture must fail with the macOS-only compile_error!. I cannot observe that from a Mac.
  • Everything else should be byte-identical in behaviour to main, since Driver::capture still calls capture_pid there.

Merging

The user told the reviewing session directly to merge in autopilot after review and green CI ("no, merge by yourself"). The Windows reviewer will merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_018qSpykxfaLiQSDxjsqbQUm

)

`shot`, `window` and every pixel assertion were Windows-only: the harness
photographs the window from outside with `PrintWindow`. On macOS it cannot.
Since #511 a harness window is never on screen, and reading another
process's pixels needs Screen Recording permission, which a test must not
ask for. So on macOS the app now takes the picture itself.

The `capture` verb renders the last drawn frame to an offscreen texture with
gpui's `Window::render_to_image`, writes the raw RGBA to
<sandbox>/suite/capture/last.rgba, and replies with the path, size, scale,
frame and `content_origin`. `uiharness` reads the file and from then on it is
an ordinary `Capture`: the origin is computed by the same `screen_rect` that
`rect` answers with, so the existing crop — a region's rect minus the
capture's origin — lands on the right pixels unchanged, and every border
probe runs exactly as on Windows. The pixels travel beside the control
channel rather than through it; a full window is megabytes of RGBA.

That method only exists behind gpui's test-support, so it needs a build made
for it: the new `harness-capture` feature. Two things about it are not
obvious and both are load-bearing:

  - It forwards to `gpui_platform/test-support`, not `gpui/test-support`.
    The macOS implementation is gated on `gpui_macos`'s own feature, so
    enabling only gpui's builds cleanly and then answers every capture with
    "not implemented for this platform".
  - It is a compile error off macOS. test-support also makes gpui draw every
    dirty window as it flushes effects, without presenting; on Windows that
    would let a frame wait be satisfied by a frame that never reached the
    screen, and `PrintWindow` would photograph the one before it.

⚠️ So a macOS pixel case tests a test-support build, whose draw scheduling
differs from the shipped app's, and a render of gpui's scene rather than what
a compositor showed — the native window buttons are absent, so `window` is
content only where `PrintWindow` includes the frame. Never ship the feature.

A default build answers `capture` with an error naming the feature, rather
than handing back a blank picture.

Measured on macOS 27 (M1) against the pinned gpui, through the real runner:
the unmodified `sheet-selection.uit` passes all five cases, including its
three border assertions (solid teal, dashed teal, solid #2f6fdb) with the
thresholds tuned on Windows left untouched. The window server reports the
window as never on screen and the frontmost app never changed. The instance
exits 0 with nothing on stderr, so leak detection raised nothing at quit.

Cost of the feature: binary 21,702,768 -> 22,137,088 bytes (+2%), twelve
more crates (proptest and its helpers), 252 s for a clean build into its own
target directory. Cargo.lock only gains packages; the gpui pin is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018qSpykxfaLiQSDxjsqbQUm
@yeroo

yeroo commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner Author

Windows review of 3decb6a:

  • cargo check --manifest-path suite/Cargo.toml -p docxy --features harness-capture on Windows fails as intended with the compile_error: "the harness-capture feature is macOS-only: it enables gpui test-support, whose draw-without-present would make Windows captures read stale pixels".
  • Driver::capture still takes capture_pid on every non-macOS target, and a default build answers capture with an error naming the feature, so the Windows harness path is unchanged.
  • Diff reviewed: sandbox-only pixel file, content-origin mapping through the same screen_rect as rect, blank/wrong-size refusal, all unit-tested. Looks good; merging when CI is green.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.87755% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
uiharness/src/driver.rs 0.00% 4 Missing ⚠️
uiharness/src/capture.rs 97.87% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@yeroo
yeroo merged commit b43316e into main Sep 28, 2026
11 checks passed
@yeroo
yeroo deleted the issue-506-macos-offscreen-capture branch September 28, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants