sous

The sous-chef for Claude's kitchen: delegate mechanical, volume-heavy coding tasks from Claude Code / Claude Desktop to a local MLX model on your Mac. Claude designs the menu; sous does the prep — in a sandboxed, auditable, autonomous tool loop. The sandbox is application-level, not an OS jail (what that means). You (and Claude) review everything it cooks.

Why

Heavy Claude Code use runs into plan usage limits, and most of what consumes them is generated output. sous exists to stretch that budget: the mechanical, volume-heavy output comes from a local model at zero marginal cost, while Claude spends its much cheaper input-side attention writing instructions and reviewing the resulting diff. Claude stays the head chef, and the same plan carries further into the week.

This is a hybrid local + cloud arrangement, deliberately not an all-or-nothing switch to a local model: pointing Claude Code itself at a local endpoint trades away the frontier reasoning you are paying for, while sous offloads only the small, mechanical work your Apple silicon Mac can handle on its own.

How sous compares

Other ways to put local models next to Claude Code make a different trade:

• Full-local replacements point Claude Code (or a fork of it) at a local endpoint — every task drops to local-model quality, including the ones you wanted a frontier model for.
• Routers/proxies swap models per request, but the work still runs synchronously inside your session, and the only thing between the local model and your files is a permission ruleset calibrated for a frontier model's judgement.
• Subagent/skill delegates hand tasks to a local model with your session's permissions and none of the rest: no persistent queue, no project-root confinement, no allowlist scoped to a model you trust less, no audit trail.

sous is the hybrid: Claude keeps the reasoning, and an asynchronous queue hands the mechanical work to a local worker that is sandboxed, budgeted, approval-gated, and journaled — with the diff always reviewed before it counts.

The cost is capability, and it is deliberate. The worker runs its own loop over eight fixed tools (read_file, write_file, edit_file, list_dir, glob, grep, run_command, finish); it cannot reach your MCP servers, your skills, or your hooks, all of which a subagent delegate inherits. sous buys confinement, persistence, and a reviewable diff by giving the local model a much smaller world to work in. Work that genuinely needs the harness is work to keep in Claude.

Requirements

• Apple silicon Mac (tested on M-series with 64 GB; the default model needs ~18 GB free unified memory — for 16 GB machines, see Smaller machines)
• Python 3.14 (standard build) via uv
• Claude Code or Claude Desktop with a Pro/Max plan

Install

uv tool install sous-mcp   # PyPI package name; the CLI it installs is `sous`
sous install-launchd       # start at login, keep alive (recommended)
claude mcp add --transport http sous http://127.0.0.1:8383/mcp

From a checkout, uv tool install . works instead of the PyPI package. (The bare sous name on PyPI is an unrelated, abandoned placeholder — the package you want is sous-mcp.)

Claude Desktop

claude_desktop_config.json launches MCP servers as stdio subprocesses, so it cannot take the HTTP URL above. Use sous mcp, which bridges stdio to the daemon:

{
  "mcpServers": {
    "sous": { "command": "sous", "args": ["mcp"] }
  }
}

sous mcp holds no state and loads no model — it forwards messages to the one daemon. Open several clients and they share it, so the model is resident once no matter how many are connected. If no daemon is running it starts one, which is what makes this work without sous install-launchd.

Discovery needs no extra setup: the daemon publishes MCP server instructions that clients put in front of Claude, saying when to delegate and why. Optionally install the delegation skill as a supplement — a fuller playbook (mirroring delegations into Claude's task list, approval etiquette, restarting a downed daemon):

cp -r skills/delegating-to-local ~/.claude/skills/

First delegation downloads the model (~16.1 GB for the default) — one time.

Managing the daemon

sous status             # is it up, and what has it been doing
sous top                # watch it live (alias: sous status --watch); q quits
sous statusline         # one line for Claude Code's status bar (below)
sous wait <task-id>     # block until a task finishes or needs approval
sous claude             # Claude Code with local subagents (gateway mode, below)
sous stop               # stop it (see below)
sous uninstall-launchd  # stop it starting at login, and remove the agent
sous tune --quick       # measure this machine and propose settings (see Tuning)

sous stop deliberately refuses when launchd is managing the daemon, because KeepAlive would restart it a second later and the command would look like it did nothing. It tells you which command you actually want. Stopping is for daemons nothing is supervising — including one sous mcp started for you.

Stopping the daemon also ends any running sous mcp bridges; their clients reconnect and start a fresh one on the next call.

The daemon writes one log, ~/.sous/daemon.log, both streams. Every line sous emits reads 2026-09-10T19:26:14.025Z INFO sous.gateway: … — UTC timestamp with milliseconds, then INFO (served or forwarded), WARNING (a request sous refused: a 4xx, a 529, a client gone while queued, tools it had to drop) or ERROR (a failure sous produced), then which part spoke. Library lines (mcp.…, uvicorn.error, huggingface_hub…) take the same shape, and warnings.warn arrives as WARNING py.warnings. Earlier releases split stdout and stderr into daemon.log and daemon.err.log; re-run sous install-launchd once — it boots the old job out, waits up to 30 s for its daemon to free the lock, folds daemon.err.log onto daemon.log (or only removes the old name when it was already a link to daemon.log), removes it, writes the new plist and bootstraps it. That re-run needs the daemon stopped first if it was started by hand (sous serve, the normal case on a machine with no launchd job installed yet): finding the lock still held, the command refuses and writes nothing — sous stop, then retry. Once the old job is unloaded, a failure before the new one loads (a daemon still exiting after the wait, a fold or plist write that fails) says the job is unloaded and exits nonzero; fix the cause and re-run. A failed launchctl bootstrap after the plist is written prints the bootstrap command to run by hand and exits nonzero rather than claiming success.

What Claude gets

Tool	Purpose
delegate_to_local_model	queue a self-contained task (returns immediately)
task_status	poll progress / queue position / approval requests
task_result	fetch report, changed files, verify output, diff
cancel_task	stop a queued or running task
respond_to_command_request	approve/deny a non-allowlisted command
server_status	engine, live turns, queue and config — the status document, minus the recent lists GET /sous/status adds

Gateway mode (experimental)

sous can also stand between Claude Code and api.anthropic.com: with [gateway].enabled = true the daemon serves Anthropic's Messages API on the same 127.0.0.1:8383, answers requests for the model id sous-local with the local model, and forwards everything else — the main loop's requests, the startup probe, usage and telemetry calls — to the real API untouched. That is the hybrid in issue #41: a frontier main loop on your subscription, Task-tool subagents on the local model, one ANTHROPIC_BASE_URL, and a frontier model always reviewing the local model's work.

sous claude                            # Claude Code, subagents served locally
sous claude -p "summarize README.md"   # every argument passes through to claude

sous claude asks the running daemon for its effective gateway settings — the config file may have been edited since it started, and only a restart applies it — and refuses if no daemon answers, if the gateway is off, or if the daemon predates routing. It says so when the file and the daemon disagree, and uses the daemon's values. Then it replaces itself with claude, having set:

Variable	Value	Why
ANTHROPIC_BASE_URL	http://127.0.0.1:8383	one endpoint; the gateway routes on the requested model id
CLAUDE_CODE_SUBAGENT_MODEL	sous-local (the first local_models entry the daemon reports)	the default model for Task-tool subagents; the main loop keeps its claude-* id
CLAUDE_CODE_SUBAGENT_MODEL_FORCE	1	the override: since Claude Code 2.1.26x a built-in agent's own model: (Explore, for one) or a per-spawn model beats the default above; this applies the default to every subagent regardless
CLAUDE_CODE_MAX_CONTEXT_TOKENS	the daemon's max_context_tokens	Claude Code has no built-in size for sous-local; it honours this variable only for non-claude-* ids, so the main loop is unaffected
API_TIMEOUT_MS	3000000	a cold model load plus a long prefill takes minutes

plus --disallowedTools LSP unless you pass your own --disallowedTools (a language server connecting mid-session appends its schema to every request and re-prefills the conversation). It sets no ANTHROPIC_AUTH_TOKEN, ANTHROPIC_API_KEY or ANTHROPIC_DEFAULT_*_MODEL: either credential variable switches Claude Code from your subscription login to API-credit billing, and the tier variables would pull the main loop onto the local model. If your shell already exports a credential variable, sous claude warns and launches anyway — that is your billing decision, not sous's (the same for an inherited ANTHROPIC_DEFAULT_*_MODEL, which would pull that tier off the upstream). It also leaves CLAUDE_CODE_AUTO_COMPACT_WINDOW alone: that setting is global, and pinning it to the local window would make the frontier main loop compact far too early; the subagent's window is bounded by CLAUDE_CODE_MAX_CONTEXT_TOKENS instead.

Before it execs, sous claude asks the daemon to keep the model loaded for the session: it POSTs /sous/hold with its own process id and start time, and since exec keeps the process id, the holder is the Claude Code process. If nothing is loaded the daemon starts loading right away, on its own thread — the launcher prints sous claude: preloading <model>; held while this session runs (or already loaded) and does not wait, so the first subagent turn finds the weights resident or the load already under way instead of paying it. The daemon's idle sweep checks its holders on every poll between delegated tasks: when the claude process exits, the hold goes with it at the next sweep — within a second on an idle daemon; a delegated task in flight defers the sweep until it finishes — and the idle clock restarts from that moment, so quitting and relaunching inside [model].idle_unload_minutes never reloads either. A hold the daemon refuses is a warning, not a stop. Two caveats: a daemon restart forgets its holders (the model then unloads after idle_unload_minutes and the next subagent turn reloads it); and sous claude --help, -h, --version and -v skip the hold, while any other invocation that exits at once (sous claude mcp list, say) holds like a session — the load it may start is one nobody waits for, and the weights then stay resident for a fresh idle_unload_minutes.

The preflight itself is plain HTTP. The daemon's own routes live under /sous/ on the same port, loopback-only like the gateway's, whether or not the gateway is on:

• GET /sous/status — one JSON document: engine (loaded, loading, unloading, model_id, idle_seconds, holders, memory_gb, the prompt_cache counters, and on the VLM backend positions, the load line's engine|model), inflight (the turn the model is serving right now — its msg_ id, phase, tokens so far, rate and ETA — usually empty or one entry, ordered with the turn on the pass first and then the queue in arrival order), queue (delegated task counts), recent_turns (the last 50, every field of the turn line below), recent_tasks (the last 10) and config. The MCP server_status tool returns the same document without the two recent_* lists.
• GET /sous/events — the same document as a Server-Sent Events stream: once at connect, then whenever something on it changed — the turn in flight (at most ten times a second), a load, an unload, a hold or its release, a task, an edit of the config file — and, only while a turn is in flight, a delegated task running or a load or unload under way, at least once a second; idle, nothing but a ping every 10 s. The idle clock is not refreshed between frames: a client advances engine.idle_seconds from the time since the frame arrived, the way sous top does; memory_gb likewise waits for the next frame.
• POST /sous/hold — what sous claude posts (above).

A 404 from /sous/status means the running daemon predates this CLI (the route did not exist); sous claude says so and exits 1 — restart the daemon from the same install (sous stop, then sous serve or sous install-launchd). There is no compatibility mode: the CLI and the daemon ship as one package.

sous top (or sous status --watch) is that stream in a terminal — open it beside a sous claude session and watch a subagent's order go from the rail to the plate — a perforated slip with a timer dial, a pixel chef whose face is the state, the walk-in's counters, and the recent orders:

sous top at 100×30, mid-decode with a second order queued — the render the test suite pins, so the picture is always the current one.

q, Esc or Ctrl-C leave the screen exactly as it was; enter opens the order under the cursor — its ticket, with the turn's numbers; l the legend, ? the About card, m turns the motion off. The vocabulary is glossed on the screen itself (SEAR ▐███▌ prefill, REHEAT 41 hit) and on its legend line; the numbers are the turn line's, in the terminal's own foreground, and nothing paints over the terminal's background. It reconnects with backoff if the daemon restarts and says so if none is running. It is the one command in sous that imports Textual; nothing else (the daemon included) loads it. It needs a terminal: piped or run over a session with no pty, it prints a message on stderr and exits 2 instead of painting the pass into a pipe and waiting for a key that cannot come.

sous statusline prints one line for Claude Code's statusLine setting — sous: decode 612 tok · 14.7 tok/s · eta 18s during a turn, sous: idle · 5 slots · held between turns, sous: daemon down when nothing answers — and reads (and ignores) the JSON Claude Code pipes to it. Add to ~/.claude/settings.json:

{"statusLine": {"type": "command", "command": "sous statusline", "refreshInterval": 1}}

refreshInterval matters: without it Claude Code re-runs the command only on session events — a new assistant message, a compaction, a mode change — so the line goes quiet for a whole subagent turn; with it the command runs every second as well. The command loads none of the heavy dependencies (no Textual, httpx or psutil); it drains stdin and fetches the status on a thread joined to a half-second budget, printing sous: daemon down if that runs out, so it costs the status bar nothing.

To bracket a subagent from the outside as well, a SubagentStart / SubagentStop hook can append its own record — {ts, agent_id, agent_type} — to a file; it sees the subagent's start and end, not the progress-summary calls Claude Code makes in between, which only the turn line and sous top show.

Forwarding is a plain HTTP/1.1 pass-through to [gateway].upstream_url (default https://api.anthropic.com): the request body goes up byte for byte; Authorization, anthropic-beta, anthropic-version and every header sous does not recognise travel unmodified; only Host and the hop-by-hop headers change — and, on the two Messages routes whose body sous had to read, Content-Length is recomputed from the exact bytes — and responses carry the upstream's headers minus the hop-by-hop set, plus a Via: 1.1 sous, with Date and Server being the daemon's own rather than the upstream's. sous stores no credential — it never sees a token it did not receive to forward — honours no HTTPS_PROXY or ~/.netrc, follows no redirect, and retries nothing (Claude Code retries). When the upstream is unreachable, forwarded requests get a 502 (504 on timeout) and the local model keeps working. WebSocket features (voice) are not forwarded.

A whole-session-local run — every tier pinned to sous-local — remains possible for exercising the endpoint; the recipe is in CONTRIBUTING.md. It is a verification setup, not a mode: it is exactly the trade "Why" and "How sous compares" above argue against.

Claude Code executes the tool calls itself, with its usual permission prompts; the local model only decides what to call. What a locally served turn gives up, stated plainly:

• No sandbox. The gateway returns tool_use blocks and never runs a tool; toolexec.py (path confinement, allowlist, audit) is not in this path. Claude Code's own permission system is the boundary, and a 27B model inherits whatever permissiveness you configured for frontier subagents.
• No Anthropic server-side or built-in tool types (no client-supplied schema). WebSearch, WebFetch-as-server-tool and code execution run inside Anthropic's API; bash_*, text_editor_* and the other built-ins run on the client but arrive with their schema implied by the type. Both are dropped from a locally served request (logged as dropped N tool(s) with no client-supplied schema) — the local chat template can only offer a tool it has an explicit schema for. Claude Code sends custom-typed equivalents when it drives a non-claude model, so a local subagent keeps its file and shell tools. The main loop, forwarded upstream, keeps everything.
• No thinking, no request-level sampling. thinking, temperature, top_p, top_k, stop_sequences and tool_choice are accepted and ignored; the daemon's [model] sampler applies. Images and documents in messages become a one-line [image omitted: sous serves text only] placeholder.
• One turn at a time; keyed prompt-cache slots. Local turns are serialized behind the same lock as delegated tasks. The prompt cache keeps one slot per resident conversation (bounded by [model].prompt_cache_gb), so a subagent's consecutive turns reuse their own slot — copied and left in place when the budget can hold the copy, so a conversation that Claude Code branches (its progress-summary calls for a background agent take the same prefix down a different last turn) still finds it, and moved into the extending turn when it cannot, which is every hit at prompt_cache_gb = 0; a linear conversation holds at most its current and previous lengths — two subagents interleaving reuse theirs, and a delegated task running in between no longer wipes the gateway's slots the way a shared single slot did. Reuse stays owner-scoped — a slot is used only by the thread that built it, so a delegated task can never adopt a gateway slot or vice versa — but the budget and memory-pressure eviction below apply across the daemon, so a delegated task's own slot can displace a least-recently-used gateway one (and at prompt_cache_gb = 0, where only one slot fits, it will). A new subagent starts from a fork when it can: a copy of an earlier conversation's cache taken at a boundary its own prompt shares. Two boundaries are kept, each when it is long enough to clear the 4096-token floor. The tools boundary is where the template's tool block ends — Qwen3.5/3.8 render the # Tools block before the client's system text, and Claude Code's tool array is byte-identical across sessions and projects for a given subagent type (Explore, general-purpose, …), so a new claude process's first subagent turn starts ~45–56K tokens warm instead of paying ~170 s cold. The header boundary is where the whole system block ends, so a same-type subagent inside the same session starts ~57K tokens warm and prefills only its own brief. Each fork is a full copy of the KV at its boundary (~3.5 GiB at 57K tokens on the default model): one tools fork per tool set, plus one header fork per session that has used it, so several live claude sessions accumulate more than the tool-set count alone suggests. Budgeted by [model].prompt_cache_gb below. Forks live as long as the weights: [model].idle_unload_minutes drops them with the model. Two subagents still run one at a time; batching is a later phase.
• Usage is split the way Anthropic's is. cache_read_input_tokens is what the turn served from a resident cache slot and input_tokens the rest, so a warm subagent turn shows a few hundred input tokens and ~57K cache reads. message_start carries the whole count (it is sent before the cache decision); message_delta and the non-streaming body carry the split, which is where the SDKs read the input-side fields from when present. No cache_creation_input_tokens: every prompt stays resident, so it would only double-count the uncached tokens.
• Mid-conversation system messages become <system-reminder> blocks. Claude Code delivers attachments that arrive after the first turn (agent listings, MCP instructions, deferred tools) as a role: "system" message after the preceding user message. The chat template takes one system turn, at index 0, so the gateway renders such a message the way Claude Code's own fallback for models without the feature does — a <system-reminder> block in the user turn before it (after that turn's tool results). The model sees the same text in the same place a frontier model without the feature would; nothing the model needs is lost, and the conversation stays a strict extension of the previous turn's, which is what keeps it warm.
• A shorter window, and Claude Code compacts inside it. A frontier subagent has a 200K-token window; a local one has [gateway].max_context_tokens (131072 by default), which sous claude passes as CLAUDE_CODE_MAX_CONTEXT_TOKENS. Claude Code auto-compacts a conversation when its own token count reaches the window minus its output reserve (20K) minus a 13K buffer — 98K at the default window — and its "precompute" variant fires earlier still, at a fraction of that it takes from remote configuration. In practice a general-purpose subagent compacts after four or five tool turns at 131072: one call to write the summary (a warm hit, but a ~3.4K-token generation at the model's decode speed — about 3.5 minutes on the default model) and one to continue from it (the summary plus every re-attached file, ~30K tokens of prefill from the tools fork), and the agent then reports from a summary of itself. Nothing on the sous side changes this — both calls are served as cheaply as their content allows — and none of Claude Code's compaction switches (CLAUDE_CODE_AUTO_COMPACT_WINDOW, DISABLE_AUTO_COMPACT, autoCompactEnabled) is per-model: each would also change the frontier main loop, so sous claude sets none of them. The lever is the window: the default model's native context is 262144, and max_context_tokens = 262144 moves the classic threshold to ~229K at the cost of one more window of KV reserved out of the auto prompt-cache budget (8 GiB on the default model — the ~27 GiB auto budget described under prompt_cache_gb below becomes ~19 GiB on a 64 GB machine, still room for a tools fork beside a retaining conversation). Where the earlier precompute trigger lands at that window is not known until measured: its fraction comes from Claude Code's remote configuration, keyed by window size. Restart the daemon after the edit; sous claude passes the running daemon's value.
• A client that disconnects does not stop the model. A local turn runs to completion (so the next request never waits on a wedged lock); aborting mid-generation comes with batching, later. A forwarded stream, by contrast, is closed upstream the moment the client hangs up.

Each /v1/messages turn served locally logs one metadata-only line, for example (wrapped here):

2026-09-10T19:26:14.025Z INFO sous.gateway: POST /v1/messages id=msg_… model=sous-local
  stream=1 status=200 input_tokens=84335 output_tokens=2887 stop=end_turn
  cache=hit took=turn@82647 reused_tokens=82647 prefilled_tokens=1681 forks=0 evicted=1 pressure=0
  load_s=0.0 queue_s=2.5 engine_wait_s=0.0 tokenize_s=1.1 ttft_s=9.8 prefill_s=7.9 decode_s=196.6
  prefill_tps=212.8 decode_tps=14.7 seconds=207.4 tools=1b21cd75 system=9f8e7d6c

id is the response's message id (the same one in message_start, so the line joins to Claude Code's transcript). cache is hit (this conversation's own slot), fork (a copy of a shared boundary — ~45–56K reused tokens is a tools fork, ~57K a header fork) or miss; took names the slot and its length — turn@N for this conversation's own slot copied and left in place, turn-moved@N for one the budget could not hold a copy of, or whose copy failed (removed and extended in place — every hit at prompt_cache_gb = 0), fork@N for a shared boundary, none on a miss, and on a hit whose warm attempt failed and was rebuilt cold (a WARNING py.warnings: … retrying cold line comes first), where prefilled_tokens and the phases below describe that cold attempt. prefilled_tokens is what the turn had to prefill; forks/evicted/pressure are what it published and what was dropped under it — evicted counts every drop — budget, pressure, or the lengths below the slot a turn takes, which the take retires — and pressure is the subset of those evicted the pressure valve forced (not a second, disjoint count). load_s (a model load, ≈0 when resident — including one this turn only waited out, started by another request), queue_s (the wait for the gateway lock — Claude Code's small background calls hold it too), engine_wait_s (the wait for the engine itself, which a delegated task's generation holds — part of seconds and ttft_s, in no phase below), tokenize_s, prefill_s and decode_s say where the time went, as far as the prompt cache measures it — not a strict partition of seconds: a turn that starts from a copied fork slot times that copy into neither phase, so the phases can sum to less than the total. ttft_s is not one more slice alongside them: it spans from the start of generation to the first token, overlapping prefill_s and however much of decode_s ran before that token, so adding every field this way can just as easily run past seconds as fall short of it. prefill_tps/decode_tps are that same phase attribution expressed as a rate, not a throughput measurement — prefill_s also carries fork copies and the snapshot, so a turn that publishes forks reports a lower prefill_tps than its real prefill speed. seconds is the total, running from the moment the turn takes the gateway lock, so client-visible latency is seconds plus queue_s. On a miss the line adds lcp= (how many leading tokens the render shared with the closest resident slot), lcp_region= (tools below the tools boundary — the tool array changed; system below the header — the system text did; tools-or-system below the header when it is the only boundary, since a tool block under the floor, or one a template renders inside the system block, cannot be told apart from the text there; conversation otherwise; - when the probe found no boundary at all, whether for lack of fork budget or because nothing cleared the 4096-token floor, or when the session held no slot to compare with, lcp=0) and bounds=[tools,header], the boundaries the probe verified — either one is left out when it never cleared the 4096-token floor, so this can also read bounds=[57123] (only the header boundary) or bounds=[] (no probe ran). tools= and system= are 8-hex-character hashes of the rendered tool array and system text: comparable across lines, not reversible. Refused requests log the same way at WARNING with status= and seconds= — but there seconds is measured from request receipt, not from the gateway lock, so it already is client-visible latency, with nothing to add. A streaming turn that fails after its SSE headers already went out logs status=200 — the status the client actually received — with an error= naming the failure; alerting on status=5.. alone misses these, so watch error= on streamed turns too. Once a request parses, its 529, 499 and failure lines carry id= too — on a streamed failure, the id message_start already delivered. A client that disconnects mid-turn does not stop the turn, and its line is still written once the turn drains. A client that disconnects while its turn is still queued behind another logs status=499 error=abandoned the same way — a local 499, distinct from the forwarder's own synthesized 499 for a client gone mid-forward (below). A locally served count_tokens logs its input_tokens, load_s (the model load it paid for), count_s (time inside the runner) and seconds (client-visible, from request receipt — it includes any wait for a free worker that count_s does not); the engine logs model_load seconds=N.N model=<model_id> when it loads, plus positions=engine|model on the VLM backend (which side supplies the rotary positions behind a warm cache). One more line names the Anthropic tool types a turn dropped, when any. Each forwarded request logs one line too: upstream, method, path, the model id when the body named one, the upstream's status, and seconds to its headers — at INFO whatever the status, since that is the upstream's verdict; a 502/504/499 the forwarder itself produced (unreachable upstream, a timeout, a client gone) logs at ERROR or WARNING instead. The daemon also disables uvicorn's access log, which would otherwise print every request target — query string included — at INFO. Nothing else is logged — not a request body, not a header value, not a query string, not a response — at any level. Errors sous produces itself are Anthropic-shaped ({"type": "error", "error": {"type": ..., "message": ...}}): an oversized body (over 32 MiB) on /v1/messages is a 413 request_too_large, a prompt that fills the local window an invalid_request_error saying prompt is too long, an unreachable upstream a 502 api_error. Errors from the real API come back exactly as it sent them.

Configuration — ~/.sous/config.toml

[server]
port = 8383

[model]
id = "mlx-community/Qwen3.8-27B-4bit"
idle_unload_minutes = 30   # a live `sous claude` session pins the model; the clock
                           # restarts when its last one exits
max_context_tokens = 32768
prompt_cache = true
# Cache slots kept resident beyond the running turn: "auto" sizes them
# from free Metal memory, a number sets the GiB, 0 keeps a single slot.
prompt_cache_gb = "auto"
temperature = 0.7
top_p = 0.8
top_k = 20
# Speculative decoding: ~1.8x decode on the default model with the shipped
# sampling, up to ~2.4x greedy. "" disables it. Block size 3 measured best
# on an M5 Pro (+3% on prose, +13% on code re-emission over the drafter's
# adaptive policy); 0 lets that policy pick the depth; anything above 5 is
# clamped, because mlx's fused attention kernel takes at most 5 verify rows
# on this model and 6–8 rows run 5–6x slower per layer. Auto-disables with
# a warning when the drafter can't serve the configured model.
speculative_draft_id = "z-lab/Qwen3.8-27B-DFlash2"
speculative_block_size = 3
# `sous tune --quick` may rewrite speculative_draft_id, speculative_block_size
# and max_context_tokens (and [gateway].max_context_tokens) after showing you
# the diff; it never changes id or int8_prefill.

# INT8-activation prefill on the M5 tensor units (M5-family or newer, macOS
# 26.2+; warns once and prefills stock elsewhere). ~1.4x prefill on the default
# model, but int8 activations change numerics — measured drift is inside what
# 4-bit weights already add — so it stays off until a tool-loop A/B says
# otherwise.
int8_prefill = false

[budgets]
max_turns = 40
max_minutes = 15
max_tokens_per_generation = 4096

[commands]
allowlist = ["pytest", "python -m pytest", "npm test", "npx eslint",
             "npx prettier", "ruff", "black", "mypy", "go test",
             "cargo test", "cargo check", "make test", "uv run pytest",
             "uv run python -m pytest", "uv run ruff", "uv run black",
             "uv run mypy", "uv run ty"]
timeout_seconds = 120
approval_timeout_minutes = 10

[context]
mode = "fixed"     # "auto": size the window per task from free memory
fraction = 0.8     # auto: share of remaining memory headroom the KV cache may use
min_tokens = 8192  # auto: never shrink the window below this

[tasks]
retention = 200

[gateway]
# EXPERIMENTAL — see "Gateway mode" above. Serve Claude Code subagents from the
# local model and forward everything else to the real API, on the same port.
enabled = false
local_models = ["sous-local"]   # model ids served locally; every other id is forwarded upstream.
                                # Never claude-*: Claude Code ignores its context-window
                                # env vars for those ids (the config rejects them).
upstream_url = "https://api.anthropic.com"  # where non-local requests go: an https origin, no path,
                                            # ASCII hostname or IP literal.
                                            # Plain http is accepted for a loopback host only.
max_context_tokens = 131072     # server-side limit on prompt + reply tokens for local turns;
                                # a Claude Code subagent's prompt — its agent prompt plus the
                                # session's tool schemas — is ~50-58K tokens before it does
                                # anything, and it asks for 32K of output; 65536 was too small
                                # in practice. Positive values below 49152 are raised to it.
                                # `sous claude` sets CLAUDE_CODE_MAX_CONTEXT_TOKENS to the
                                # running daemon's value of this (restart it after an edit);
                                # without the launcher, set it yourself or a long subagent
                                # conversation grows past it and fails with "prompt is too long".
                                # Claude Code auto-compacts a local subagent once its own
                                # count nears this minus ~33K (sooner with its precompute
                                # trigger); 262144 — the default model's native length —
                                # pushes that out at the cost of 8 GiB of prompt-cache budget.
generation_timeout_minutes = 30

Every value is optional; the allowlist is re-read on every command execution, so edits apply instantly. Swap [model].id for any MLX text or vision model (e.g. the -8bit/-4bit conversions, or a fast MoE coder via mlx-lm).

[context] mode = "auto" sizes the worker's context window per task instead of using the fixed [model].max_context_tokens: when a task starts, sous measures the remaining memory headroom (the tighter of the Metal working-set ceiling and available system RAM), lets the KV cache have fraction of it, and clamps the result between min_tokens and the model's native maximum. The window is a cap, not a reservation. With cache reuse on (the shipped default), the KV cache lives for the whole task, so fraction bounds sustained residency, and a run_command subprocess competes with a live cache that used to be freed between turns. With [model].prompt_cache = false it bounds only a per-generation peak. Residency still tracks the tokens a task actually uses, not the window. Every task's report records the window it ran with and why (budget.context_tokens / budget.context_reason). The default model's hybrid attention makes context unusually cheap (only 16 of its 64 layers accumulate KV — about 64 KiB per token), so an otherwise-idle 64 GB machine gets the full native 262k window. If sizing fails for any reason, the task runs with the fixed max_context_tokens and a warning. Cache reuse pays most in auto mode: since elision is the only thing that discards the cache, and elision fires only when the prompt exceeds the window, a window the task never reaches means the cache survives the whole task. The shipped default is fixed at 32768 tokens.

[model].prompt_cache (default true) reuses a KV cache across the turns of a conversation, prefilling only what the conversation gained instead of the whole thing every turn. All of a task's generations run on one worker-owned thread so the cache survives between turns; measured on the default model in one process, six growing turns took 29.5s warm against 77s cold, with per-turn time flat instead of growing. Set it to false to prefill every turn from scratch.

[model].prompt_cache_gb (default "auto") bounds the caches kept resident beyond the turn that is running: each conversation's current length, its previous one when the budget holds both (see below), plus fork slots at every boundary long enough to be worth copying (4096 tokens or more): one at the end of the tool block, shared by every Claude Code session and project that presents the same tool array, and one at the end of the whole system block, shared by same-type subagents of one session (see above). Prefixes must match token for token — one added, removed or reordered tool is a different tool set with its own tools fork and header forks. Each fork is a copy of the KV at its boundary, ~3.4–3.6 GiB at ~57K tokens on the default model: one tools fork per tool set, plus one header fork per session that has used it, so a daemon that has seen the usual three tool sets across one or two live claude sessions holds roughly five to nine forks, ~17–31 GiB (the tools forks stay most-recently-used, since every new session touches them). "auto" is what Metal's recommended working set has left once the weights, one full context window of KV (the larger of [model]'s and [gateway]'s) and 2 GiB of slack are paid for — about 27 GiB on a 64 GB machine with the default model and gateway window, room for those forks and several conversations; each live conversation under the auto budget also keeps its previous length resident (the slot a branch of it starts from), so a live conversation is two slots, ~8 GiB at 63K tokens, beyond the forks; a 48 GB machine should set it to 0 (forks off, one slot), or to more than twice one conversation slot at the length its conversations reach — a little over 8 GiB with the default model at ~57K tokens — because the copy that keeps a conversation's previous length resident is taken only when that length and the longer one the turn publishes both fit; below that every hit moves its slot, and a branch of the conversation (a progress-summary call) starts from the fork instead. The copy's room comes from least-recently-used slots, forks included, so keeping a fork resident beside a retaining conversation wants about four slots' worth, ~14 GiB. A value below ~14 GiB makes every cold turn take a fork copy that a later turn then evicts to make room for its own copy, so it pays the fork and never reuses it. Slots are evicted least-recently-used first when the budget, a count of 16, or memory pressure says so. Pressure is two readings: Metal's own headroom (room for one more window of KV), and the kernel's memory-pressure level — at warn each publish drops one least-recently-used slot (a cold turn publishes up to three times: two forks and its turn slot; a warm turn once), at critical every slot but the one that just ran. The kernel's level is used rather than free RAM because a freshly loaded model leaves ~17 GB of its weight files in the page cache, which reads as "used" for a while and would evict the forks a cold turn had just made. The conversation that just ran is never evicted by its own turn, and a cold turn's second fork copy never evicts its first, so 0 means exactly one slot (the pre-3a behaviour) and a 32 GB machine degrades to that on its own. Forks live as long as the weights do: idle_unload_minutes drops them with the model, so "every new session" means every new session inside that window. server_status (and GET /sous/status, the same document over HTTP, plus the recent turns and tasks) reports the engine — holders (live sous claude sessions pinning the model), loading (a load in progress, a preload included), memory_gb and prompt_cache — slots, resident bytes, hits, fork hits, retained and moved turn-slot takes, evictions and the subset the pressure valve took — counts only — and inflight, the turn being served right now with its phase, tokens and rate.

[model].int8_prefill (default false) runs the prefill matmuls as INT8 activations against the checkpoint's packed 4-bit weights on the M5 GPU's neural accelerators (Apache-2.0 kernel derived from oMLX, compiled at model load — no build step). Measured on an M5 Pro with the default model: 492 → 695 tok/s at 4K tokens, 410 → 570 tok/s at 32K (MLP and linear-attention projections; attention projections are not routed, see #76). Decode and speculative verify are untouched. It changes prefill numerics (KL 0.033 vs the stock path on a code prompt; 4-bit weights alone are 0.052 vs 8-bit), which is why it ships off. Needs an M5-family or newer GPU and macOS 26.2+; anywhere else the server_status tool reports int8_prefill: unavailable with the reason and prefill runs stock. Only dense Qwen3.5-family models (model_type qwen3_5, as the default model is) with affine 4-bit, group-size-64 weights route, and the MoE variant is refused; a checkpoint with no eligible projection warns once; in a mixed checkpoint, ineligible projections fall through per projection.

temperature/top_p/top_k control the worker's sampler (Qwen's own documented non-thinking-mode defaults). Greedy decoding (temperature 0) sounds safer but isn't: it gives the model no way to escape a bad completion once it happens, since a near-identical prompt plus a nudge still argmaxes to the same wrong output every time.

Tuning

sous tune runs its quick stage only in this version — pass --quick, or it refuses and says so. It measures this machine and proposes the [model] settings that cannot change what the model says: the drafter, its block size, and a context window that fits. It detects the chip, the Metal working set and whether the GPU has tensor units, fits every curated candidate to memory (and prints the arithmetic for each one it refuses), lists every download it would need and asks about each one separately, then measures prefill and decode throughput of every arm through sous's own engine (--repeat sets how many attempts each decode and short-prefill measurement gets, default 2 — the best attempt wins; the one 16K prefill is the prefix its decode continues from) — the numbers a delegated task or a gateway turn would see. It ends with a report, a diff of ~/.sous/config.toml, and a question:

Apply these changes to ~/.sous/config.toml? [y/N]

Nothing is written before that yes; --apply applies the diff without asking (downloads are still asked about individually unless --yes, which answers every prompt for scripted use). A backup is kept beside the config file. The daemon is asked to release the model first (POST /sous/unload) and refuses while a sous claude session holds it, a task is running or queued, or a load or unload is under way — the tune waits for none of them, it tells you, and it asks once more right before its first model loads. Applied changes take effect only when the daemon next starts — sous stop, then sous serve (or launchctl kickstart -k gui/<uid>/<label> for a managed one) — and the tune prints that exact command when a change needs it. Results and the report land under ~/.sous/tune/<run-id>/; --resume <run-id> continues an interrupted run, --models ID ... measures ids of your own. Other models than the configured one are measured and reported with a "quality untested" label; only the full run (a later release) may propose a model change.

Smaller machines

sous tune --quick tells you what fits and how fast it runs here; the table below is the fallback for a machine that cannot reach the Hub. The alternative shares the default's qwen3_5 architecture, so it loads through the exact same mlx-vlm path — edit [model].id in ~/.sous/config.toml and the next delegation downloads and uses it.

Unified memory	[model].id	Weights
64 GB / 32 GB (default)	mlx-community/Qwen3.8-27B-4bit	~16.1 GB
16 GB	mlx-community/Qwen3.5-9B-MLX-4bit	~6 GB

The default is the affine 4-bit Qwen3.8-27B: half the footprint of the 8-bit-class quants with no measured tool-loop quality loss. It also keeps mlx-vlm's speculative-decoding fast path available, which requires affine quantization (4-, 5-, or 8-bit); mxfp quants fall into a much slower per-token verify fallback (#58 has the measurements behind both claims). The 16 GB pick drops to the 9B tier because an 8-bit 9B (~11 GB) would crowd the ≈10.7 GB Metal working-set limit of a 16 GB machine once the KV cache lands on top of the weights; on 16 GB, also consider [model].max_context_tokens = 16384 if you see memory pressure. (mlx-community/Qwen3.5-9B-mxfp8/-mxfp4 look like the obvious picks, but as of 2026-08 they are empty placeholder repos with no weights.)

The 16 GB alternative passed the same worker-path validation as the default (see Validation status), run on the 64 GB test machine — which validates the models and quants through sous's whole stack, not the memory fit on physical 32 GB / 16 GB hardware (that remains arithmetic: weights plus KV-cache headroom). Smaller workers still fail more tasks in general and make reviewing the diff matter more — but a reviewed draft from a small local model costs your plan nothing.

Security model

• Workers are confined to the project_root of their task (symlink-resolved; .git/ writes denied). The sous control directory (~/.sous/ — config, allowlist, task db, transcripts) is never writable from inside the sandbox, and a project_root that contains it is rejected outright.
• No shell: commands run as argv (never through a shell), with an environment scrubbed down to PATH, HOME, LANG, LC_ALL, TERM, TMPDIR (anything else, including *_TOKEN/*_KEY/*_SECRET/*_PASSWORD vars, is stripped). Only allowlisted commands run without approval. PATH itself is adopted from your login shell once at daemon startup, so allowlisted commands resolve exactly as they do in your terminal no matter how the daemon was launched (launchd starts agents with the bare system PATH, which would otherwise turn every uv run ... into an approval).
• Non-allowlisted commands pause the task for explicit human approval (auto-deny after approval_timeout_minutes).
• Path confinement bounds the worker's edits, not what an allowlisted command can do. Allowlisting a command that executes repo-resident code — any test runner (pytest, npm test, make test, ...) — is equivalent to granting arbitrary local code execution over code the worker just wrote: the worker can write a conftest.py or test file that does anything the command's process can (network egress, reading any user-readable file), and the allowlisted verify run executes it without approval. Calibrate the allowlist accordingly, and review diffs before trusting verify output.
• There is no network sandbox: an allowlisted or human-approved command can still reach the network if the command itself does (e.g. npm test hitting a registry). Keep the allowlist narrow and review approval requests before saying yes.
• Command timeouts kill the command's whole process group (SIGTERM, short grace, SIGKILL) before file changes are audited — but a descendant that double-forks and calls setsid() escapes into a new session and survives the group kill. Closing that residual requires cgroup/OS-level confinement that macOS does not offer.
• The before/after file audit around each command is stat-based: (mtime_ns, size, ctime_ns) per file. mtime and size alone are forgeable by code the command executes (equal-length rewrite + os.utime restore); ctime is what makes the audit tamper-resistant, because no userspace API can set it and os.utime itself bumps it. That resistance has limits: a process running as root (e.g. via a mount trick or raw-device write) or manipulation of the system clock between the two snapshots could still hide a change. The audit is a safety net against the sandboxed worker and the code it runs — not against a privileged attacker.
• Every worker turn is journaled to ~/.sous/tasks/<id>/transcript.jsonl.
• The MCP endpoint binds to 127.0.0.1 only.
• Gateway mode bypasses the sandbox by design. A locally served Claude Code turn never touches toolexec.py: the gateway hands tool_use blocks back and Claude Code executes them under its own permission rules. The gateway binds to 127.0.0.1 only and refuses foreign Host/Origin values, and any browser request whose Sec-Fetch-Site is not none or same-origin (a page's <iframe> or no-cors GET carries no Origin to refuse), on every route, forwarded ones included. It forwards the Authorization header Claude Code sends with every request to [gateway].upstream_url unmodified and nowhere else, stores it nowhere, adds no credential of its own (no ~/.netrc, no proxy environment), and never logs a request body, header value or query string. A plain-http upstream is accepted for a loopback host only. It is off by default.

Validation status

Validated end to end on an M5 Pro / 64 GB (originally against mlx-community/Qwen3.8-27B-mxfp8, which remains a supported [model].id):

• Worker path — a delegated "add type hints and docstrings" task completed in 57s over 3 turns (done / completed). The worker edited the file, chose to run pytest to check itself, and reported accurately; an independent re-run of the tests confirmed it.
• MCP surface — driven by a real MCP client over streamable HTTP, the same path Claude Code uses: all six tools registered with the expected names, and a delegated task ran to done / completed in 42s with a correct diff and verify output.
• Current default (Qwen3.8-27B-4bit) — three delegated tasks (module-from-spec, docstring sweep, test scaffolding) through the real worker loop on 2026-08-29, all done / completed in 4/7/3 turns; every artifact passed independent grading, including hidden spec tests (#58).
• Other models through the same stack — the same worker-path check also passed on Qwen3.8-27B-mxfp4 in 35s over 4 turns — confirming mlx-vlm loads mxfp4-mode quants — and on the 16 GB pick Qwen3.5-9B-MLX-4bit in 25s over 8 turns.

Tool-call parsing accepts both the XML-ish <function=…>/<parameter=…> format that Qwen3 emits and the hermes JSON format used by other MLX models.

Limitations

• MLX generation cannot be aborted mid-stream. Generations are serialized by a per-engine lock (and the engine is never idle-unloaded while one is in flight), so a truly wedged generation delays subsequent tasks until the daemon is restarted. Running the worker in a separate process (process isolation) is the future fix.
• scripts/e2e_smoke.py uses a 0.6B model so it stays cheap to run. That model is too small to reliably emit a finish call, so the script usually ends failed or budget-exhausted even when it writes the right file — it exercises the plumbing, not model competence. The real-model runs above are the meaningful end-to-end evidence.
• Gateway mode (experimental) serves one local turn at a time on a keyed prompt cache (a conversation's last two lengths plus tools and header forks, budgeted by [model].prompt_cache_gb), drops Anthropic server-side and built-in tool types from local turns (the ones that carry no client-supplied schema), ignores request-level sampling and thinking, and finishes a local turn even after the client hangs up. It serves the model ids in [gateway].local_models locally and forwards everything else to [gateway].upstream_url over HTTP/1.1 only — no WebSocket upgrade, no HTTP/2, no proxy environment.

Development

Setup, the checks CI runs, and the pull-request process are in CONTRIBUTING.md.

Manual E2E with the real model: sous serve, register with claude mcp add, then ask Claude to delegate something trivial and watch sous status.

Design spec: docs/superpowers/specs/2026-08-14-sous-design.md.