feat(automation): pointer, keyboard, hover, scroll, drag and wait in the app window

.opencode/skills/codenomad-automation/SKILL.md

@@ -8,7 +8,7 @@ description: Inspect, modify, rebuild, relaunch, and verify CodeNomad desktop bu
 Use this workflow when changing CodeNomad itself:
 
 1. Call `codenomad.inspect` before editing. It identifies the local CodeNomad window and returns an indented accessibility outline with refs, plus runtime diagnostics. Large parts collapse with `(+N hidden)`/`(+N more)`: pass `ref` to expand one, or `query` to find elements by text or role.
-2. Use refs from the latest inspections with `codenomad.act`: `click`, `type` (replaces the field's text; `clear: false` appends) and `select` (native `<select>` options by label or value). Each result returns the field value or chosen options to check. Inspect again after navigation, reload or failed actions.
+2. Use refs from the latest inspections with `codenomad.act`: `click` (a real pointer click, refused when the element is covered), `type` (replaces the field's text; `clear: false` appends), `select` (native `<select>` options by label or value), `hover`, `drag` onto another ref, `scroll`, `press` a key or chord such as `Enter`, `Escape`, `Shift+Tab` or `Mod+K`, and `wait` for text to appear or disappear. Results return the field value, chosen options or focused element to check. Inspect again after navigation, reload or failed actions.
 3. Make the smallest source change and run the relevant typecheck, native test, or build. Automation does not own OpenCode: the shared `opencode2` daemon and this session remain alive if CodeNomad closes.
 4. After updated artifacts are ready, call `codenomad.act` with `action: "restart"`. The call gracefully relaunches the host pinned by the latest inspection, waits for the same artifact identity to reconnect to the OpenCode session, and returns a fresh inspection. It never switches to another worktree's build.
 5. Call `codenomad.screenshot` for final visual evidence and report the checks that ran.

AGENTS.md

@@ -156,7 +156,7 @@
 
 - One bundled `codenomad.automation` V2 plugin uses native discovery and backend presence. All browser/developer tools are available without a Developer Mode toggle; native instrumentation starts with the desktop host. Loading, tool availability and execution targeting are separate: retain the authenticated bridge and session/window fences. Tauri preview children receive no application capabilities; primary-renderer reload must dispose them and final-window checks must count native windows. See `dev-docs/BROWSER_AUTOMATION.md`.
 - Developer UI tools target the inspected native window/run, not the currently selected project or conversation. Keep backend ownership discovery separate from UI focus; agents must be able to inspect and navigate back from another session themselves. Accessibility refs expire on document/target replacement, not merely on session selection. Browser previews retain their own session attachments.
-- Page automation (outline, refs, click/type/select) lives once in the transport-agnostic `packages/server/src/automation/page/` engine; inspections stay budgeted plain-text outlines that announce what they collapse, never a silent cut. In the CodeNomad window page functions carry `__codenomadAutomationDepth`, clicks stay programmatic and native text insertion yields only input events, so automation can never produce the trusted dock click/key press a human Form answer requires: never add unmarked trusted input or arbitrary script evaluation there. See `dev-docs/BROWSER_AUTOMATION.md`.
+- Page automation (outline, refs, click/type/select/hover/drag/scroll/press/wait) lives once in the transport-agnostic `packages/server/src/automation/page/` engine, with tool input validated by its `parse.ts` on both sides of the bridge; inspections stay budgeted plain-text outlines that announce what they collapse, never a silent cut. Pointer actions aim at the visible centre and refuse covered elements. In the CodeNomad window page functions carry `__codenomadAutomationDepth` and every trusted DevTools input the expiring `__codenomadAutomationInputUntil` lease, both read by `lib/automation-input.ts`, so automation can never produce the trusted dock click/key press a human Form answer requires: never add unmarked trusted input or arbitrary script evaluation there. See `dev-docs/BROWSER_AUTOMATION.md`.
 
 - Desktop automation and pruning provisioning follows the authenticated OpenCode connection and its `config.get` global discovery directory, including reconnects. Never derive a running daemon's roots from backend/startup environment or CLI `debug paths`. OpenCode watches the entire discovery root recursively: keep changing presence leases outside it, in the sibling `.codenomad/<root-hash>/` namespace. Retain existing outside-root storage; migrate inside-root storage while reading older backends' leases without writing there. WSL translates the daemon-reported paths through the selected distro only for filesystem access. Verify heartbeat stability with the isolated native automation fixture.
 - Plugins register only at their next presence check, so a backend's first plugin RPC could arrive before registration and be refused. Followers acknowledge a lease with a `<lease>.ready` marker (current namespace only) once every follower of that directory in the native process has observed it while registered; routes calling bundled-plugin RPCs await `workspaceManager.desktopPluginsReady()` before reading or authorizing the session, so ownership checks and deletion-fence entry stay adjacent (bounded to 5 s, never rejects). The bound is paid in full only when no marker can arrive: no Location loaded yet, or pre-marker plugin copies still loaded after an upgrade. Do not replace this ordering with retries of refused reads.

dev-docs/BROWSER_AUTOMATION.md

@@ -20,13 +20,17 @@ The bridge remains loopback-only, token-authenticated, bounded, and session-loca
 
 ## Page engine
 
-`codenomad.inspect` and `codenomad.act` (the CodeNomad window) run on the shared page engine in `packages/server/src/automation/page/`, which only needs a DevTools protocol channel: `snapshot.ts` renders the accessibility tree, `actions.ts` acts through fixed page functions, and `refs.ts` names DOM nodes. Browser-preview tools still use their native controllers.
+`codenomad.inspect` and `codenomad.act` (the CodeNomad window) run on the shared page engine in `packages/server/src/automation/page/`, which only needs a DevTools protocol channel: `snapshot.ts` renders the accessibility tree, `actions.ts` performs actions, `input.ts` dispatches trusted pointer, wheel and keyboard input, `keys.ts` maps key chords, `parse.ts` validates tool input on both sides of the bridge, `protocol.ts` holds the shared protocol primitives and `refs.ts` names DOM nodes. Browser-preview tools still use their native controllers.
 
 - Inspection is an indented outline with refs, read as plain text. Wrappers, unnamed images and text repeating its parent's name are dropped; fields show their value instead of their inner nodes; native selects list their options with `selected`/`disabled`. A character budget (32 000 by default) is shared among siblings: small parts stay whole while large ones collapse in place with `(+N hidden)` or `(+N more)`, keeping their first and last items, so nothing is cut silently. `ref` expands one subtree and `query` keeps elements whose name, value or description contains it, or whose role is exactly it, with their contents and ancestors.
 - A ref names one node within one target, run and navigation epoch. It keeps its name and stays usable for the life of that page generation, whatever is inspected later; a node that left the page fails its action explicitly.
 - `type` focuses the field and refuses one that cannot take focus (hidden, disabled, behind a modal dialog), because native text insertion goes to the focused element. It then replaces the field's text by default (select, then native insertion, so framework input handlers fire); `clear: false` appends. Date, time, range and color inputs are validated before anything changes, then set with `input`/`change`, keeping the browser's normalization; email and number inputs append as a value. `select` picks native `<select>` options by label, else value, and dispatches `input`/`change` only when the choice changes; clicking an option ref picks that exact option. Results return the field value (passwords masked, rich text with its line breaks) or the chosen labels.
-- Trust rule: in the CodeNomad window page functions run under `__codenomadAutomationDepth` and clicks stay programmatic. Native text insertion is unmarked but produces only input events, which the human-answer check never accepts: it requires a trusted click or key press on the dock's submit (`packages/ui/src/lib/form-answer-interaction.ts`). Never add trusted CDP mouse or keyboard input to the application window without a mark, and never expose arbitrary script evaluation there.
-- `packages/ui/tests/browser/automation-forms.test.ts` drives a real Chromium page and the real Mission editor through `DeveloperCdp`.
+- `click` is a real pointer click at the centre of the element's visible part, refused when another element would receive it (hit testing respects `pointer-events: none`, and a label of the target counts as the target); an element without a visible box is activated programmatically. `click` on a native select asks for `select` instead of opening its out-of-page popup. `hover` and `scroll` (wheel on a ref or the page) use the same aiming. `drag` requires both ends already visible, without scrolling under a saved start point; it uses stepped pointer moves for pointer-based drag libraries (HTML drag-and-drop is not emulated). `press` dispatches a key or chord (`Enter`, `Shift+Tab`, `Control+K`, `Mod+K` = Meta on macOS) to the focused element, optionally focusing a ref first and refusing it unless focus actually moved there, and returns the newly focused element. Failed input sequences best-effort release held buttons/keys on their original channel without replaying the action. `wait` polls (≤ 15 s) for text in the rendered page or in labels, titles, placeholders and values, or for its disappearance. Native application-menu accelerators handled by the host may not see dispatched keys.
+- Trust rule: in the CodeNomad window page functions run under `__codenomadAutomationDepth`, and every trusted DevTools input (pointer, wheel, keys, text insertion) under the short `__codenomadAutomationInputUntil` lease in `performance.now()` units. Ending it retains its end timestamp: `lib/automation-input.ts` checks the originating event's `timeStamp`, not when a stalled renderer handles it. Delayed automated keys and their same-turn implicit submit clicks remain automated after expiry. `form-answer-interaction.ts` refuses human provenance for those events, so automation can never produce the trusted dock click or key press a human Form answer requires. Never add unmarked trusted input to the application window, and never expose arbitrary script evaluation there.
+- An input timeout has an **unknown outcome**, not a proven failure: a busy renderer can deliver an already admitted key or click later, including when cleanup releases its held input. Cleanup still attempts release when its marker write times out, but never repeats the press. The tool explicitly instructs inspection before retrying; never automatically retry these inputs.
+- The marker returns the page's own epoch timestamp for CDP mouse/key events. Cleanup retains the original press timestamp even if its marker renewal times out, so delayed releases remain within the original provenance interval independently of command-timeout settings. Missing/invalid provenance timestamps refuse before input dispatch.
+- When a Preferences or remote window has focus, window automation targets the most recently used application window, as the update flow does; the Preferences window itself is not an automation target.
+- `packages/ui/tests/browser/automation-forms.test.ts` drives a real Chromium page and the real Mission editor through `DeveloperCdp`; `automation-input.test.ts` covers trusted input and proves that automation clicks and Enter in the real InterruptionDock never carry the human-answer header, while a real user click does.
 
 ## September 2026 architecture decision
 

packages/electron-app/electron/main/main.ts

@@ -191,9 +191,11 @@ function runPrimary(firstIntent: LaunchIntent) {
   const developerMode = new DeveloperMode({
     devtoolsDataPath: sessionDataPath,
     nativeIdentity: developerNativeIdentity,
+    // A focused Preferences or remote window is not an automation target: fall back to the
+    // most recently used application window instead of reporting no target at all.
     targetWindowId: () => {
       const focused = BrowserWindow.getFocusedWindow()
-      return focused ? registry.resolve(focused.webContents)?.id : registry.mruRecord()?.id
+      return (focused ? registry.resolve(focused.webContents)?.id : undefined) ?? registry.mruRecord()?.id
     },
     requestRelaunch: () => lifecycle.requestRelaunch(),
   })