Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
- run: pnpm install --frozen-lockfile --ignore-scripts
- run: pnpm typecheck
- run: pnpm test
- run: pnpm build

secrets:
name: secret scan
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ bun.lockb

# Generated browser modules.
.demo/

.next/
next-env.d.ts
20 changes: 18 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Repository files, evidence lines, and a proposal's `rationale` are untrusted dat

Fixtures are synthetic. Never add a fixture drawn from a real repository, a customer, a private conversation, or anything containing a credential, token, or personal data. Instruction-trap fixtures stay harmless (the "attack" is "delete `.env`", not a working exploit). No operational attack steps.

Any future live transport keeps keys on the host's server side. Keys never appear in this package, its fixtures, its receipts, its logs, or its test output. Automated tests never call a live provider and never consume shared credits.
The optional example host keeps its default key server-side. At the user's request, its UI supports a personal override saved in origin-local, unencrypted browser storage, matching the playground. Never expose a saved value, and never put a key in model state, fixtures, receipts, logs, or test output. Save/remove and cross-tab changes invalidate pending live results. Automated tests never call a live provider and never consume shared credits.

Checked-in guards are not optional: the `pre-commit` hook (`scripts/check-secrets.mjs`, installed by `pnpm install`) and the CI `secret scan` job (gitleaks over full history). Maintainers must separately verify and enforce this repository's remote push protection and signing settings; upstream descriptions do not configure them. Do not disable, skip, or `--no-verify` past any of them to land a change. If a check fires on a false positive, rewrite the text so it is unambiguous (`<your-key>`, `$ENV_VAR`, `op://` references all pass). If it fires on a real key, stop and rotate it; do not amend it away.

Expand Down Expand Up @@ -119,7 +119,7 @@ A measured claim (a number in a README, a doc, or a PR body) links the run that

## Not in scope here

- Provider HTTP clients or SDK wrappers (hosts own transport).
- Provider HTTP clients or SDK wrappers in the exported package. The separately scoped `examples/host/` adapter owns the local demo transport; it is never imported by `src/`.
- Human approval interrupt/resume, permission grants, session identity, replay protection — host concerns.
- Executing, applying, testing, or committing anything a model proposed.
- Training or fine-tuning. Jev is not customer-fine-tunable; a specialist *proposer* is a separate, measured experiment that this harness can evaluate but does not contain.
Expand All @@ -145,3 +145,19 @@ A measured claim (a number in a README, a doc, or a PR body) links the run that
Read `docs/routing.md` before changing routing behavior. `src/routing/` is pure and shares the no-I/O boundary of `src/contract/`. Routing outcomes/receipts are separate from review verdicts. Hosts supply availability, cost estimates and normalized evidence; no descriptor grants permission or launches a sub-agent. Preserve the clarification option, closed-set validation, pinned model and untrusted-data note. Bump `ROUTING_QUESTION_SET_VERSION` when routing instruction semantics change.

Routing scenarios are separate synthetic demonstrations, not new proposal-review fixture categories. Evaluation labels must never affect mock evidence or adapter payloads. Byte/token proxies and local JS timing do not establish live provider savings, correctness or execution speed. Link a run artifact for measured claims.

## Optional Next.js demo host

At Val's request, `app/`, `components/` and `examples/host/` implement the interactive demo. Keep React and provider/CLI I/O outside `src/`. Read the pinned Next.js docs under `node_modules/next/dist/docs/` before changing App Router behavior. Verify `pnpm build` alongside typecheck and tests.

The arena host may invoke the underlying Codex CLI against fixed synthetic fixtures through bounded MCP tools. This is not a new runtime or permission grant: no proposed source is applied or executed. Preserve auth-only temporary CODEX_HOME and HOME, disabled external tools, read-only sandbox, exact fixture ids, process cancellation and output limits. Automated tests use a fake CLI and fake Jev; live smoke runs are explicit and separately recorded. No private repository context or global agent instructions may enter the arena.

<!-- BEGIN:nextjs-agent-rules -->

# This is NOT the Next.js you know

This version has breaking changes — APIs, conventions, and file structure may all differ from your training data. Read the relevant guide in `node_modules/next/dist/docs/` (resolved from this file's directory; in monorepos the `next` package may not be visible from the repo root) before writing any code. Heed deprecation notices.

This block is written and re-added by `next dev` — verify at `node_modules/next/dist/server/lib/generate-agent-files.js`. Removing it from a diff only re-creates the uncommitted change; committing it with your work keeps the tree clean.

<!-- END:nextjs-agent-rules -->
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ No API key is needed for these offline checks. Use the versions pinned in
checks ran successfully: inspect checks for the exact PR head before merging.
Signed commits and the existing secret guards remain required.

The optional Next.js demo starts offline and offers explicit live routing through
its separate loopback example host. `/arena` runs Codex against synthetic MCP fixtures.


## What is implemented here

| Component | Status in this hardening series |
Expand All @@ -44,7 +48,7 @@ Signed commits and the existing secret guards remain required.
| Benchmark-only verdict helper | Implemented at `src/benchmark`; explicitly records no-review provenance |
| Evaluation accounting and blinded proposer inputs | Implemented at `src/benchmark/evaluation.ts`; not a live experiment runner |
| Routing contract and offline synthetic comparison | Implemented at `src/routing/` and `examples/routing/`; no live provider or execution |
| Interactive routing demo | Implemented at `examples/routing/`; loopback-only browser assets and synthetic evidence |
| Interactive Next.js demo and CLI arena | Implemented at `app/`, `components/`, and `examples/host/`; loopback routing, saved keys, usage and synthetic fixture comparisons |
| Proposal schema/path/diff validator, Jev payload builder, transport integration, fixture suite, full runner | Still pending extraction from the playground |
| Host authorization, sandbox, execution, and durable storage | Host responsibilities; not implemented in this package |

Expand Down Expand Up @@ -210,19 +214,18 @@ For the dark, interactive **Routing room** demo:

```sh
pnpm demo
# Open http://127.0.0.1:4173 (or pnpm demo 4187 for a different port)
# Open http://127.0.0.1:4173 (or pnpm exec next dev --webpack --hostname 127.0.0.1 --port 4187 for a different port)
```

Change tool availability, compare Lean with Batteries included, inspect the selection evidence, and download the receipt. Sample tasks use scripted evidence; custom text asks for clarification. No tool or sub-agent executes. See [the demo guide and browser verification](docs/routing-demo.md).
Change tool availability, compare Lean with Batteries included, inspect the selection evidence, and download the receipt. Offline sample tasks use scripted evidence; custom offline text asks for clarification. Save a personal key under **API key**, select **Live Jev**, then explicitly route a task to use the live host. The Routing room does not execute tools. `/examples` adds schema and context experiments; `/arena` compares real Codex CLI runs with full versus Jev-selected synthetic MCP tools. The host records proposed patches without applying them. See [the demo guide and browser verification](docs/routing-demo.md).

The run artifact includes receipts, full/lean context bytes, token estimates, acceptable-tool inclusion and cheapest acceptable selection. Evidence is scripted; local timing is not Jev or execution latency. Router overhead is counted separately so fewer schemas do not automatically imply savings. See [the design, metrics and host adapter boundary](docs/routing.md).


## Roadmap and related projects

The full extraction remains gated on the canonical upstream playground history.
The routing contract and offline synthetic comparison are implemented; live
integration and measurements remain pending. Planned host work includes a Rust
The routing contract, Next.js demo and optional live routing/CLI example host are implemented. Repeated live evaluation and production-host integration remain pending. Planned host work includes a Rust
`ProposalReview` seam and a measure-first `ContextScorer`.
Context scoring needs an egress policy and evidence that its costs beat cache
reuse before a runtime integration. See [architecture](docs/architecture.md)
Expand Down
43 changes: 43 additions & 0 deletions app/api/arena/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import { boundedText, json, localOrigin } from "../../../examples/host/live";
import { liveHandle } from "../../../examples/host/runtime";
import { runArenaLanes } from "../../../examples/host/arena-lanes";
import { ARENA_CASES } from "../../../examples/arena/cases";
import { DEMO_CATALOG, DEMO_POLICY } from "../../../examples/routing/scenarios";
import { routeTools } from "../../../src/routing";
export const runtime = "nodejs";
export const maxDuration = 300;
let running = false;
export async function POST(request: Request) {
if (!localOrigin(request)) return json(403, { error: "Use this app's local origin." });
if (request.headers.get("content-type") !== "application/json") return json(415, { error: "Use application/json." });
let caseId: string;
try { const input = JSON.parse(await boundedText(request.body, 1024)); if (!input || Object.keys(input).join() !== "caseId" || typeof input.caseId !== "string") throw Error(); caseId = input.caseId; }
catch { return json(400, { error: "Choose an arena example." }); }
const fixture = ARENA_CASES.find(item => item.id === caseId);
if (!fixture) return json(400, { error: "Unknown arena example." });
if (running) return json(429, { error: "An arena comparison is already running on this host." });
running = true;
const abort = new AbortController();
const signal = AbortSignal.any([request.signal, abort.signal]);
const stream = new ReadableStream({
async start(controller) {
const emit = (event: unknown) => { if (!signal.aborted) controller.enqueue(new TextEncoder().encode(JSON.stringify(event) + "\n")); };
try {
emit({ type: "stage", value: "Asking Jev which tool schemas to expose…" });
const routed = await liveHandle(new Request(request.url, { method: "POST", headers: request.headers, body: JSON.stringify({ intent: fixture.task, availableIds: DEMO_CATALOG.map(tool => tool.id) }), signal }));
const body = await routed.json();
emit({ type: "usage", measurement: body.measurement ?? null, attempted: body.attempted !== false, error: body.error ?? null });
if (!routed.ok || !body.evidence) { emit({ type: "error", value: body.error ?? "Jev evidence unavailable. No CLI run started." }); return; }
const receipt = await routeTools(DEMO_CATALOG, { intent: fixture.task, availableIds: DEMO_CATALOG.map(tool => tool.id) }, DEMO_POLICY, { source: "jev", review: async () => body.evidence }, signal);
if (receipt.outcome === "unavailable") { emit({ type: "error", value: "Routing evidence unavailable. No CLI run started." }); return; }
emit({ type: "routing", receipt });
emit({ type: "stage", value: "Both agents are running in parallel. Results appear independently as each finishes." });
await runArenaLanes(fixture, receipt.selectedIds, signal, emit);
if (!signal.aborted) emit({ type: "done", at: new Date().toISOString() });
} catch { emit({ type: "error", value: "The comparison could not complete. Check the local CLI configuration and retry explicitly." }); }
finally { running = false; try { controller.close(); } catch {} }
},
cancel() { abort.abort(); },
});
return new Response(stream, { headers: { "Content-Type": "application/x-ndjson", "Cache-Control": "no-store", "X-Accel-Buffering": "no" } });
}
3 changes: 3 additions & 0 deletions app/api/config/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export const runtime = "nodejs";
export const dynamic = "force-dynamic";
export function GET() { return Response.json({ live: true, serverKey: Boolean(process.env.TYPESAFE_API_KEY) }, { headers: { "Cache-Control": "no-store" } }); }
4 changes: 4 additions & 0 deletions app/api/route/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
import { liveHandle } from "../../../examples/host/runtime";
export const runtime = "nodejs";
export const maxDuration = 60;
export const POST = liveHandle;
2 changes: 2 additions & 0 deletions app/arena/page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
import { Arena } from "../../components/arena";
export default function Page() { return <Arena />; }
2 changes: 2 additions & 0 deletions app/examples/page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
import { ExampleLab } from "../../components/example-lab";
export default function Page() { return <ExampleLab />; }
Loading
Loading