auto-routing.md

Auto routing with Jev

The same Jev transport and credential lookup also serve approval rules. Approval judgments are independent of Auto routing: they only evaluate pending approval requests and never choose or run a coding model.

Auto is the ⚡ toggle in the composer. When it is on, Modex asks a small, fast judge a few typed questions about each request before the turn runs, then picks the backend, model, reasoning effort, and fast mode for that turn — within limits you set — and shows a one-line receipt in the transcript explaining why. The coding turn itself still runs only through the claude and codex CLIs; nothing about Auto changes that.

What the judge is

The judge is Jev, TypeSafe's System One model. It does not generate text and never sees your files; it answers typed questions about a compact digest of the request and returns calibrated probabilities. Modex asks six questions in one request (apps/desktop/src/main/engine/routing/judge.ts, question set v1):

id	type	what it decides
task	choice	quick answer · small edit · bug fix · feature · refactor · investigation · review · ops · unclear
complexity	score 0–3	one-step … cross-cutting, each level a concrete situation
blast_radius	score 0–2	one file … could delete data / rewrite history / touch other systems
wants_speed	noul	the user signalled a quick turnaround matters more than thoroughness
needs_deep_reasoning	noul	doing it well needs multi-step reasoning about interactions
depends_on_prior_turns	noul	only when the thread has history: the request leans on it

The state sent is the request (clipped to 4k chars), the last six transcript entries (user text clipped, a placeholder for each assistant reply, tool titles), and thread/project facts (backend, mode, plan, turn count, project name, branch). No file contents, diffs, or tool output are sent.

Key and transport

The key can live in any of these places; Modex checks them in this order and tells you which one won in Settings → Auto routing:

1. Modex itself — paste it into Settings. It is encrypted with the OS keychain (Electron safeStorage: macOS Keychain, Windows DPAPI, or the Linux keyring) and written to ~/.modex/app/secrets.json with mode 0600. It is never in state.json, never in a build, and the renderer only ever sees the last four characters.
2. TYPESAFE_API_KEY or JEV_API_KEY in Modex's environment.
3. The jev CLI's config, ~/.config/jev/config.json (JEV_CONFIG overrides), written by jev config set apiKey … — one configuration for every tool on the machine.
4. TYPESAFE_API_KEY from your login shell, because a Finder-launched app inherits no profile.

Any of those may hold a 1Password reference (op://Vault/Item/field) instead of a literal key; Modex expands it in memory through op read at use time and reports a locked vault as the problem rather than silently judging with the heuristic.

The judge itself is reached one of two ways (Settings → Judge transport):

• jev CLI (default when installed): Modex runs jev run - --raw --compact --attempts 1 with the payload on stdin and the resolved key in the child's environment only. That gives Modex the CLI's config, its error messages, and jev doctor, and keeps one code path for people, scripts, and agents. Install with npm link in TypeSafeAI/cli.
• Built-in HTTPS (POST https://api.typesafe.ai/v1/systemone): the same request from Modex's own process, used when the CLI is absent or the policy says so.

Transport selection is explicit: Auto prefers the CLI and may fall back to HTTPS; Always the jev CLI never uses HTTPS if its executable is unavailable; Always Modex's HTTPS call does not inspect the CLI. Saving changed judge transport, executable, or model settings invalidates resolved setup without clearing learned preferences or disrupting an already-running turn. General, posture-only, and no-op saves retain connection-test health.

"Test judge" sends one tiny question through whichever is active and shows the answer or the API's own error sentence (a 402 with no credits, a 401 for a rejected key). A rejection disables Jev for the session — no per-turn retry latency — until the key changes or a test succeeds. With no key and no CLI, Auto still works on the built-in heuristic and every receipt says so.

Pricing and limits, from the TypeSafe model page at the time of writing: jev-latest → jev-1.13.0, charged per input token at $0.042 per million (output free), 32k tokens of state per request. A routing call sends a few hundred tokens, so Auto costs well under a hundredth of a cent per turn and typically answers in well under a second; a turn waits at most 8 s for the judge before falling back to the heuristic.

The decision service

Every question Modex puts to Jev goes through one module, apps/desktop/src/main/engine/routing/decisions.ts: Auto routing, the follow-up suggestion and approval rules, including Try it. It owns the key, the transport and the session-wide switch-off. An auth or billing rejection seen by any of them turns Jev off for all of them until the key or judge settings change or a test succeeds. Each caller still asks its own questions and handles its own failure: routing falls back to the heuristic, follow-ups use the heuristic pick, and approval rules ask a person.

Each kind of judgment has a versioned question set. The version goes up whenever its wording or choices change:

Set	Version	Questions
routing	1	the six above (routing/judge.ts)
follow_up	1	follow_up (routing/followup.ts)
approval	1	destructive plus one per language rule (approvals/gate.ts)

Each request to Jev leaves a receipt in ~/.modex/app/decisions.jsonl, one JSON line per judgment, written with mode 0600. A receipt records:

• the set and its version, the Jev model and transport, and when it ran and how long it took;
• the thread, or preview for Try it;
• Jev's typed answers (choice and confidence, score and confidence, or noul), or the error.

It never holds the state sent to Jev: no request text, digest, command or path. Past 4 MB the file starts over, and the previous file is kept as decisions.1.jsonl. The connection test in Settings is not a judgment and leaves no receipt. A receipt that can't be written is logged once and never changes a decision.

What code decides

Jev's answers are raw judgments. The policy (routing/policy.ts) turns them into a route, deterministically, and every step that changes the outcome adds a reason to the receipt:

1. Confidence gate. If Jev's confidence in the task kind is below min_confidence (default 0.6), Auto keeps whatever the thread was already using and says so. A blank model means the CLI's advertised default. When no effort was requested, it uses that model's default effort or medium, matched to supported levels under the active ceiling.

2. Target tier. Starts at the complexity score; +1 for deep reasoning; +1 for a wide or hard-to-reverse blast radius; a quick answer with low complexity pins tier 0; posture shifts one tier down (economy) or up (quality); the learned per-task offset is added; clamped to 0–3; capped at 2 once the daily premium budget is spent.

3. Backend. Stays put unless allow_backend_switch is on and the thread has no existing CLI session. Even then it only moves to an allowed backend when the current CLI cannot reach the target tier or is unavailable. Mid-thread switches lose the CLI's context, so this is off by default; model switches have their own guard in step 7.

4. Model. The highest rung at or below the target tier on that backend (routing/catalog.ts). The ladder is rebuilt from each backend's model list: Modex's fixed Claude aliases (haiku 0 · sonnet 1 · opus 2 · fable 3) and Codex's live catalogue (gpt-5.5 0 · gpt-5.6-* 1 · gpt-6-luna/sol 2 · gpt-6-astra 3), with description hints for anything new and tier 2 for unknowns. Successful non-empty lists are cached for a minute. A failed or empty list retries on the next turn; discovery errors appear in receipts and any resulting stop message.

5. Effort. Tier sets the base (low/medium/high/xhigh); deep reasoning bumps one; an explicit speed signal drops one; max_effort caps it; the budget caps it at high. The result snaps to an effort the model actually lists. Claude gets --effort, Codex gets the per-turn effort field. Supported-effort matching and low-confidence routes also obey the ceiling. If an Auto turn cannot establish a supported effort within the limit, it stops before coding execution with an explanation instead of silently using an unknown provider default. Once the daily premium budget is spent, Auto also stops if the only usable model is top tier or a low-confidence judgment pins a top-tier model. Pinned premium Auto routes count toward the same daily limit. The offline mock backend has no reasoning-effort dimension. Blocked turns do not consume the routing budget or teach learned preferences. Blocked receipts read "Auto blocked". A manual model repair after a block does not teach a tier preference from an older successful route.

6. Fast mode. Only when the user signalled speed, the task is not reasoning-heavy, the pick is tier ≤ 1, and the model offers it. Codex: the fast service tier for this turn only (serviceTierForTurn). Claude: --settings '{"fastMode":true}' for the session. Claude Code runs fast mode only on Opus 4.8 and newer Opus models, so only opus offers it and this step never picks it on Claude. On a subscription, it also needs usage credits turned on, and it draws on them even while plan usage remains. Claude reports in system/init and result whether fast mode is serving (fast_mode_state) and, if not, why (fast_mode_disabled_reason). When Auto asked for fast mode and Claude ran without it, the receipt drops "fast" and gives the reason, for example "Fast mode did not apply: fast mode requires usage credits (extra_usage_disabled)."

7. Holding the model. On a thread with a CLI session, a switch re-reads the whole history without the prompt cache and drops the previous model's thinking. So Auto keeps the model while the cache is likely warm. It still chooses freely on the first turn, after the thread has been idle 60 minutes or more, when the last turn left under 20k tokens of context, and when you have picked a different model since the last turn. While the cache is warm it never moves down, and it moves up only to a model two or more tiers higher, or when Jev is at least 0.85 sure the request needs deep reasoning and the request does not lean on earlier turns. Effort still follows the request. A probe on 2026-10-12 (Claude Code 2.1.296, codex-cli 0.162.1) showed an effort change keeps the whole cache on both CLIs, while a model change re-read it from scratch. A kept model runs at the effort Auto chose for the model it would have picked. A model kept a tier below that pick runs at higher effort to make up for it, and the receipt says "Raised effort from medium to high to make up for staying below …". A model kept above the pick runs at the pick's lower effort. A held turn does not reuse the thread's last effort, so a raise does not carry into later held turns. The max_effort ceiling, the budget's high cap and the model's supported levels still apply. A raise to xhigh or max counts as a premium turn, as moving up to the bigger model would have. Fast mode is held, because neither CLI's fast mode could be measured. Limits still win: a model over the premium budget or no longer listed is not held. A held route reads "Auto kept", with the cache size and idle time in its reasons.

Changing the policy never requires re-asking Jev; the raw judgments are reusable.

Learning-file writes are best-effort after a safe decision: a write failure appears in the receipt without blocking the coding turn or changing its outcome. Premium counts and learning stay in memory for the session and are included in the next successful write. They cannot survive an app restart until persistence succeeds.

The follow-up question

After a turn completes, the composer offers one next message. The candidates are fixed strings in routing/followup.ts (verify, review, investigate, plan, explain, continue), so nothing the judge says can become prompt text: it only picks a key, or none.

What leaves the process is followUpState: the request, the task kind from the heuristic, plan and mode, and four booleans about the completed turn — whether tools edited files, ran checks, whether any tool failed, and whether the answer mentions remaining work. Assistant text, tool arguments and tool output stay local; the unit test asserts the request never carries them. Jev is asked only on Auto threads, with the same confidence floor as routing; a non-Auto thread, an offline judge, a low-confidence or unknown choice, and every error fall back to fallbackFollowUp, which prefers a failed tool → investigate, plan mode → plan, "next steps" → continue, edits → review (checks ran) or verify (they did not), quick answer → explain, else continue.

The runner caches one answer per transcript state, so the renderer can ask on every render without asking Jev twice for the same turn. A new send or Stop cancels and clears it, and a turn that ended in an error or a Stop offers nothing.

How it learns you

~/.modex/app/routing-fit.json (routing/fit.ts) keeps a per-task-kind tier offset:

• You pick a different model by hand on an Auto thread → strongest signal. Choosing a higher tier than Auto did nudges that task kind up by 0.34 tiers (three consistent overrides move it a full tier); a lower one nudges down. Modex confirms what it learned in a notice.
• An Auto turn fails → a small nudge up (0.1).
• Offsets are clamped to ±1.5 and rounded to the nearest half tier when applied, so one override never flips a decision on its own.

The file also counts premium turns per day (premium_turns_per_day bound) and keeps the last 200 routing records with outcomes, which is the dataset a future calibrated fit model (or a threshold sweep in .probes/) would train on. Settings → "Reset learning" clears it.

Cache usage

Both CLIs report token usage for every model call: Claude on each assistant message (cache_read_input_tokens, cache_creation_input_tokens, input_tokens), Codex in thread/tokenUsage/updated. Modex keeps two things from each turn: how the first call's input split between cache reads, cache writes and uncached tokens, and the total input of the last call, which is what the next turn re-reads. Sub-agent calls are not counted; they have their own context.

The thread stores the latter as lastTurn in state.json, with the model, effort and fast mode the turn ran on and when it ended. Its fast mode is what the CLI reported when it says (Claude does), otherwise what Modex asked for. Auto's routing records in routing-fit.json add the idle time and context size at decision time and the turn's cache figures afterwards, so the records show what each switch actually cost.

Bounds you control

Settings has separate General, Coding CLIs, Auto routing, Approval rules, and Advanced / demo sections, with Save and Cancel kept visible while the content scrolls. Auto routing exposes posture, effort ceiling, minimum judge confidence, premium turns per day, fast-mode allowance, backend switching, and whether new threads start on Auto, plus the judge model and allowed coding backends. All of these are settings.routing in ~/.modex/app/state.json.

Ordinary settings are drafts until Save. Saving waits for persistence before publishing new in-memory settings; a failed save keeps both the active configuration and the draft available for retry. A view-refresh failure after persistence is reported as a refresh failure rather than claiming that the save failed. Cancel, Escape, and backdrop dismissal discard drafts and are blocked while an asynchronous settings action is pending. Draft edits and other actions are disabled until the current operation settles.

Credential Save/Clear and Reset learning are separate immediate actions: Cancel does not undo them. Learning reset requires confirmation. Keys remain encrypted outside state.json; the UI receives only masked key metadata.

The allowed coding-backend list governs switching destinations; it does not move an existing session or enable switching by itself. An empty list keeps the current backend. Legacy Mock entries remain available for offline routing until explicitly disabled in Advanced / demo; the menu identifies them separately from coding CLIs.

Test judge uses the saved judge configuration. If transport, executable, or judge model has an unsaved change, apply it before testing. Configured/untested, explicitly verified, and failed tests are distinct; results identify their configuration and are invalidated when that configuration changes. Opening Settings never performs a paid provider ping.

Which CLIs, and local models

Modex drives coding agents only through their CLIs, so a routing target is any CLI that Modex has a backend for. Surveyed on 2026-09-25:

CLI	headless protocol	approvals	model / effort control	verdict
Codex (codex app-server)	JSON-RPC over stdio, streaming items	server→client requests	model, effort, serviceTierForTurn per turn; live model/list with efforts and service tiers	shipped; the richest surface
Claude Code (claude -p)	stream-json in/out	control_request on stdio	--model, --effort low…max, fast mode via settings	shipped
Gemini CLI	-p --output-format stream-json	--approval-mode default/auto_edit/yolo/plan (no per-call prompt channel found)	-m	good next candidate once its permission prompts can be answered from a host
GitHub Copilot CLI	-p	--allow-all-tools only	--model (incl. auto)	possible for chat/plan modes; no interactive approval path yet
opencode (installed build)	-p -f json	none	none in headless	not a fit for agent mode
Local models	via Codex `--oss --local-provider lmstudio	ollama(LM Studio hasqwen/qwen3.5-9b` on this machine)	Codex's	Codex's

Recommendation: keep Claude and Codex as the routing targets now; add local models as a Codex profile (codex --oss) rather than a new backend, so AGENTS.md's CLI-only rule holds; evaluate Gemini CLI as the third backend when its headless approvals can be driven.

Verifying

• Unit: pnpm --filter @modex/desktop test covers the heuristic judge, the policy table, the fit arithmetic, the router with a fake Jev transport and with Jev failing, and the runner applying a route before a turn.
• End to end: pnpm test:e2e toggles Auto on the mock backend and checks the receipt.
• Live: export TYPESAFE_API_KEY, start Modex, turn on Auto, send a request; the receipt reads "Jev 0.xx". Settings distinguishes configured transport from a successful explicit Test judge result; a routing receipt alone is not a claim that all provider capabilities or future availability have been verified.