Computer-Use Sandbox

A Docker container that runs a full virtual Linux desktop with a stealth browser, and exposes an HTTP API to observe and control it. Built for driving websites, browser extensions, and desktop GUI applications programmatically — with a live view you can watch or take over via the browser.

What's inside

Layer	Component
Virtual display	Xvfb on :99
Window manager	Openbox (so real windows + extension popups work)
Live view	x11vnc + noVNC (watch/interact in your browser)
Capture	ffmpeg x11grab (screenshots + video)
Virtual input	xdotool (moves the real X pointer, injects real keys)
Browser	patchright (undetected Playwright) Chromium, headful
Captcha	Pluggable third-party solver (2captcha implemented)
API	FastAPI on :8000, supervised by supervisord

Why headful + xdotool (not headless)?

• Extensions only load in a real, non-headless Chrome profile.
• Desktop apps need an actual X display and window manager.
• Anti-bot evasion: OS-level input via xdotool moves the real pointer and posts real key events, which is far harder to fingerprint than CDP-synthetic input. patchright additionally strips common automation tells.

Quick start

cp .env.example .env        # optional: set API_TOKEN, CAPTCHA_API_KEY, BROWSER_PROXY
docker compose up --build

• API docs (Swagger): http://localhost:8000/docs
• Live desktop view: http://localhost:6080/vnc.html
• Health check: http://localhost:8000/health

If API_TOKEN is set, send Authorization: Bearer <token> on all mutating calls.

API overview

Observe

Method	Path	Description
GET	/health	Status + whether the display is reachable
GET	/screenshot	Full-screen PNG
GET	/screenshot/grid	Full-screen PNG with a labeled coordinate grid (spacing, major_every, font_size)
GET	/mouse/position	Current pointer {x, y}
POST	/video/start	Start screen recording ({fps})
POST	/video/stop	Stop; returns the file path
GET	/video/download?path=...	Download a recording
GET	/vnc	Redirect to the live noVNC viewer

Control input

Method	Path	Body
POST	/mouse/move	{x, y}
POST	/mouse/click	{x?, y?, button, clicks}
POST	/keyboard/type	{text, delay_ms}
POST	/keyboard/key	{key} e.g. "Return", "ctrl+a", "alt+Tab"

Browser

Method	Path	Body
POST	/browser/start	{extensions?: [absolute paths]}
GET	/browser/status	selected tab's url, title, and tab_id
GET	/browser/tabs	live tabs with stable IDs and selected_tab_id
POST	/browser/tabs/select	{tab_id, bring_to_front?: true}
POST	/browser/navigate	{url, wait_until}
POST	/browser/evaluate	{expression} (JS in page)
POST	/browser/stop	—

Browser API operations target the explicitly selected tab, initially the first tab when the browser starts. List tabs to discover new tabs or extension popups, then select their ID before navigating or evaluating them. Selecting a tab also brings its window to the front unless bring_to_front is false. Mouse and keyboard APIs always act on the actual desktop focus; changing focus through the desktop does not change the API's selected tab.

curl -s http://localhost:8000/browser/tabs
curl -s -X POST http://localhost:8000/browser/tabs/select \
     -H 'content-type: application/json' -d '{"tab_id":"tab-2"}'

IDs remain stable while a tab is open, and are not reused after browser restart. Unknown or closed IDs return HTTP 404. If the selected tab closes while other tabs remain, status reports tab_id: null and selection_required: true; navigation and evaluation return HTTP 409 until a live tab is explicitly selected. If all tabs close, navigation/evaluation create a fresh page. Listing tabs and checking status do not launch a stopped browser. Every /browser/* endpoint, including GET requests, requires the bearer token when configured.

Captcha

Method	Path	Body
POST	/captcha/recaptcha-v2	{site_key, page_url, timeout_s}

Examples

# Screenshot
curl -s http://localhost:8000/screenshot -o shot.png

# Open the browser and navigate
curl -s -X POST http://localhost:8000/browser/start -H 'content-type: application/json' -d '{}'
curl -s -X POST http://localhost:8000/browser/navigate -H 'content-type: application/json' \
     -d '{"url":"https://example.com"}'

# Click at a coordinate and type
curl -s -X POST http://localhost:8000/mouse/click   -H 'content-type: application/json' -d '{"x":640,"y":400}'
curl -s -X POST http://localhost:8000/keyboard/type -H 'content-type: application/json' -d '{"text":"hello world"}'
curl -s -X POST http://localhost:8000/keyboard/key  -H 'content-type: application/json' -d '{"key":"Return"}'

# Record video
curl -s -X POST http://localhost:8000/video/start -H 'content-type: application/json' -d '{"fps":15}'
# ... do stuff ...
curl -s -X POST http://localhost:8000/video/stop

Loading a browser extension

Mount the unpacked extension into the container and pass its absolute path:

# docker-compose.yml
volumes:
  - ./my-extension:/ext/my-extension

curl -X POST http://localhost:8000/browser/start -H 'content-type: application/json' \
     -d '{"extensions":["/ext/my-extension"]}'

Desktop applications

Install the app in the Dockerfile (apt or download), then launch it on the display from a shell inside the container (DISPLAY=:99 your-app &) or add a supervisord program for it. Drive it with the same /mouse and /keyboard endpoints and watch via noVNC.

Notes, limits, and cautions

• Gmail / Google login was intentionally left out of this scaffold. Google actively blocks automated sign-in; expect verification walls and lockouts. When you're ready, the realistic path is persistent profiles + human-like xdotool input + residential/mobile proxies, or reusing existing sessions.
• Captcha services may violate a target site's Terms of Service. The integration is provided as a pluggable hook; use it responsibly and legally.
• One browser, one recording at a time in this scaffold (single global session). Multi-session support would be the next extension.
• Run only against sites/apps you are authorized to automate.

Tests

With the Python requirements installed, run from this directory:

python -m unittest discover -s tests -v

The tab-targeting tests use fake pages and contexts, without starting Chromium, reading wallet credentials, or modifying the persistent browser profile.