[feature] support openrouter ori

README.md

@@ -11,7 +11,7 @@
 
 ## What
 
-BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. **A benchmark is just a frozen environment** — point BenchFlow at any of them, drive it with *any* ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.
+BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. **A benchmark is just a frozen environment** — point BenchFlow at any of them, drive it with any registered ACP agent or native harness adapter, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.
 
 ## Quick start: 1. Submit a trajectory
 

docs/concepts.md

@@ -8,7 +8,7 @@ The mental model for benchflow. Read once, then refer back from the how-tos.
 | Primitive | What it is |
 |-----------|------------|
 | **Task** | A directory on disk: a `task.md` document (YAML frontmatter + prompt body) plus `environment/Dockerfile` for the sandbox, `verifier/` checks, and optional `oracle/` — or the legacy split layout (`task.toml` + `instruction.md` + `tests/` + `solution/`). Authored once, evaluated many times. |
-| **Agent** | A registered ACP-speaking program (Claude Code, Gemini CLI, OpenCode, etc.). Identified by name (`"gemini"`, `"opencode"`) plus an optional model ID. Use the `acpx/` prefix (e.g. `acpx/gemini`) to route through [ACPX](https://acpx.sh/), a headless ACP client with persistent sessions and crash recovery. |
+| **Agent** | A registered agent program (ACP-speaking programs such as Claude Code, Gemini CLI, and OpenCode, or a native session-factory adapter such as Ori). Identified by name (`"gemini"`, `"opencode"`, `"ori"`) plus an optional model ID. ACP agents can use the `acpx/` prefix (e.g. `acpx/gemini`) to route through [ACPX](https://acpx.sh/), a headless ACP client with persistent sessions and crash recovery. |
 | **Environment** | The sandbox where the agent runs and the verifier checks the result. Docker locally, Daytona for cloud, Modal for serverless/GPU. Abstracted behind the `Sandbox` protocol — bring your own sandbox backend. |
 | **Verifier** | The test runner that scores the rollout. Its entry point is a `test.sh` script (native `verifier/test.sh`, legacy `tests/test.sh`) — which typically runs `pytest` against the workspace the agent left behind. For subjective tasks, use an [LLM-as-judge](./llm-judge.md) verifier scored against a rubric. Outputs `rewards: {reward: float}`. See the [verifier file map](#verifier-file-map) for which file lives where in native vs legacy packages. |
 | **Rollout** | One agent run on one task. Holds the lifecycle (setup → start → install → execute → verify → cleanup). All higher-level primitives below are built on Rollouts. |

docs/getting-started.md

@@ -49,19 +49,22 @@ uv sync --extra dev --locked
 
 ## Auth: OAuth, long-lived token, or API key
 
-You don't need an API key if you're a Claude / Codex / Gemini subscriber. Three options, pick one per agent:
+You don't need an API key if you're using a Claude / Codex / Gemini
+subscription or an OpenRouter login. Three options, pick one per agent:
 
 ### Option 1 — Subscription OAuth from host CLI login
 
 If you've logged into the agent's CLI on your host (`claude auth login`,
-`codex login`, or the `gemini` interactive flow), benchflow picks up the
-credential file and copies it into the sandbox. No API key billing.
+`codex login`, `ori login`, or the `gemini` interactive flow), benchflow picks up the
+credential file and copies it into the sandbox. No API key needs to be
+exported; billing and entitlements still follow the account used by that CLI.
 
 | Agent | How to log in on the host | What benchflow detects | Replaces env var |
 |-------|---------------------------|------------------------|------------------|
 | `claude-agent-acp` | `claude auth login` (Claude Code CLI) | `~/.claude/.credentials.json` | `ANTHROPIC_API_KEY` |
 | `codex-acp` | `codex login` (Codex CLI) | `~/.codex/auth.json` | `OPENAI_API_KEY` |
 | `gemini` | `gemini` (interactive login) | `~/.gemini/oauth_creds.json` | `GEMINI_API_KEY` |
+| `ori` | [`ori login`](https://openrouter.ai/docs/guides/ori/harness) | `~/.ori/credentials.json` or `~/.openrouter/credentials.json` | `OPENROUTER_API_KEY` |
 
 When benchflow finds the detect file, you'll see:
 
@@ -91,6 +94,7 @@ export ANTHROPIC_API_KEY=sk-ant-...
 export OPENAI_API_KEY=sk-...
 export CODEX_API_KEY=sk-...       # Codex alias for OPENAI_API_KEY
 export GEMINI_API_KEY=...
+export OPENROUTER_API_KEY=sk-or-... # OpenRouter / Ori
 export LLM_API_KEY=...           # OpenHands / LiteLLM-compatible providers
 export AZURE_API_KEY=...
 export AZURE_API_ENDPOINT='https://<resource>.openai.azure.com/'
@@ -163,6 +167,14 @@ GEMINI_API_KEY=... bench eval run \
 
 # List the registered agents
 bench agent list
+
+# Run OpenRouter's native Ori coding harness
+OPENROUTER_API_KEY=... bench eval run \
+  --source-repo benchflow-ai/skillsbench --source-path tasks \
+  --include citation-check \
+  --agent ori \
+  --model openrouter/anthropic/claude-sonnet-4.6 \
+  --sandbox docker
 ```
 
 `bench eval run` is the primary command for running evaluations — it works for
@@ -201,7 +213,7 @@ Each run writes under `--jobs-dir` (default `jobs/`):
       timing.json                   # per-phase timing breakdown
       prompts.json                  # prompts sent to the agent
       trajectory/
-        acp_trajectory.jsonl        # full agent trace (ACP events)
+        acp_trajectory.jsonl        # full normalized agent trace (legacy filename)
         llm_trajectory.jsonl        # raw provider requests/responses (when the usage-tracking proxy captured exchanges)
       trainer/
         verifiers.jsonl             # trainer-ready scored trajectory (Verifiers/ORS record)

docs/reference/python-api.md

@@ -271,6 +271,7 @@ result = await bf.run(config)
 | `claude-agent-acp` | ACP | ANTHROPIC_API_KEY | `claude` |
 | `codex-acp` | ACP | OPENAI_API_KEY, CODEX_API_KEY, CODEX_ACCESS_TOKEN, or host login | `codex` |
 | `opencode` | ACP | inferred from model/provider | — |
+| `ori` | Ori JSONL session factory | OPENROUTER_API_KEY or host login | — |
 | `openhands` | ACP | LLM_API_KEY | `oh` |
 | `pi-acp` | ACP | ANTHROPIC_API_KEY | `pi` |
 | `openclaw` | ACP | inferred from model | — |
@@ -282,7 +283,7 @@ as `azure-foundry-openai/gpt-5.5` or
 `azure-foundry-anthropic/claude-opus-4-5`. BenchFlow routes these providers
 through LiteLLM on both Docker and Daytona.
 
-Any agent can be prefixed with `acpx/` to run via [ACPX](https://acpx.sh/) (e.g. `acpx/gemini`, `acpx/claude`). ACPX is a headless ACP client with persistent sessions and crash recovery. The underlying agent's install, env, credentials, and skill paths are preserved.
+Any ACP-speaking agent can be prefixed with `acpx/` to run via [ACPX](https://acpx.sh/) (e.g. `acpx/gemini`, `acpx/claude`). ACPX is a headless ACP client with persistent sessions and crash recovery. The underlying agent's install, env, credentials, and skill paths are preserved. Non-ACP session-factory agents such as `ori` use their own native session mechanism instead.
 
 ## Retry and Error Handling
 

docs/running-benchmarks.md

@@ -315,6 +315,7 @@ Common choices:
 | Gemini | `gemini` | `GEMINI_API_KEY` or host login |
 | Claude Code | `claude-agent-acp` (alias: `claude`) | `ANTHROPIC_API_KEY` or host login |
 | Codex | `codex-acp` (alias: `codex`) | `OPENAI_API_KEY`, `CODEX_API_KEY`, `CODEX_ACCESS_TOKEN`, or host login |
+| OpenRouter Ori | `ori` | `OPENROUTER_API_KEY` or `ori login` |
 | OpenHands | `openhands` (alias: `oh`) | `LLM_API_KEY` |
 | Harvey LAB harness | `harvey-lab-harness` (alias: `harvey-lab`) | Provider key matching model |
 
@@ -324,7 +325,7 @@ Foundry models use `AZURE_API_KEY` plus `AZURE_API_ENDPOINT` with prefixes such
 as `azure-foundry-openai/gpt-5.5` or
 `azure-foundry-anthropic/claude-opus-4-5`.
 
-Any agent can also be run via [ACPX](https://acpx.sh/) by prefixing with `acpx/`:
+Any ACP-speaking agent can also be run via [ACPX](https://acpx.sh/) by prefixing with `acpx/`:
 
 ```bash
 bench eval run --tasks-dir tasks/edit-pdf --agent acpx/gemini --model gemini-3.1-flash-lite-preview --sandbox daytona
@@ -333,6 +334,36 @@ bench eval run --tasks-dir tasks/edit-pdf --agent acpx/gemini --model gemini-3.1
 ACPX is a headless ACP client that adds persistent sessions and crash recovery.
 The underlying agent's install, env vars, credentials, and skill paths are all preserved.
 
+### OpenRouter Ori harness
+
+`--agent ori` runs Ori's built-in coding harness directly through its headless
+JSONL runtime. BenchFlow pins and verifies the Ori binary, resumes Ori's native
+session id for follow-up turns, normalizes its messages and tool calls into the
+standard trajectory, and records token usage from Ori's terminal event.
+
+Use an OpenRouter API key:
+
+```bash
+export OPENROUTER_API_KEY=...
+bench eval run \
+  --tasks-dir tasks/citation-check \
+  --agent ori \
+  --model openrouter/anthropic/claude-sonnet-4.6 \
+  --sandbox docker
+```
+
+The first `openrouter/` selects BenchFlow's provider; the remainder is the
+OpenRouter model slug sent to Ori. For OpenRouter Auto, use
+`--model openrouter/openrouter/auto` (also Ori's registry default).
+
+Alternatively, install Ori on the host, run `ori login`, and unset
+`OPENROUTER_API_KEY`. BenchFlow detects either `~/.ori/credentials.json` or
+`~/.openrouter/credentials.json`, copies it into the sandbox, and uses Ori's
+native login. API-key runs go through BenchFlow's LiteLLM usage gateway; native
+login runs use the trusted token totals in Ori's JSONL terminal event.
+
+Ori is not an ACP server, so do not prefix it with `acpx/`.
+
 The **Harvey LAB harness** agent is special — it runs Harvey LAB's own agent loop
 (6 tools, system prompt) inside BenchFlow's sandbox. Use it for parity testing
 (same agent on both original and converted tasks).

src/benchflow/agents/env.py

@@ -388,8 +388,8 @@ def uses_native_subscription_auth(
     """Return True when an agent should use CLI/subscription auth directly.
 
     This is the Harbor-style split point: API-key runs can be routed through
-    LiteLLM, while subscription-auth runs stay on the native Codex/Claude ACP
-    path and report usage from the agent protocol response.
+    LiteLLM, while subscription-auth runs stay on the agent's native auth path
+    and report usage from the agent protocol/runtime response.
     """
     if agent_env.get("BENCHFLOW_PROVIDER_NAME") == "litellm" or any(
         agent_env.get(key) for key in _LITELLM_RUNTIME_MARKER_KEYS
@@ -414,6 +414,27 @@ def uses_native_subscription_auth(
             or check_subscription_auth(agent, required_key)
         )
 
+    # Registry-driven OpenRouter-login gate. Ori accepts credentials created by
+    # `ori login` from ~/.ori or ~/.openrouter and reports trusted usage in its
+    # terminal JSONL event, so it can bypass LiteLLM when no API key is present.
+    openrouter_cfg = AGENTS.get(agent)
+    if (
+        openrouter_cfg is not None
+        and openrouter_cfg.subscription_auth is not None
+        and openrouter_cfg.subscription_auth.replaces_env == "OPENROUTER_API_KEY"
+    ):
+        if agent_env.get("OPENROUTER_API_KEY"):
+            return False
+        if model is not None:
+            from benchflow.agents.registry import infer_env_key_for_model
+
+            if infer_env_key_for_model(model) != "OPENROUTER_API_KEY":
+                return False
+        return (
+            agent_env.get(_SUBSCRIPTION_AUTH_MARKER) == "1"
+            or check_subscription_auth(agent, "OPENROUTER_API_KEY")
+        )
+
     # Registry-driven Claude-CLI gate: any agent whose subscription_auth
     # substitutes ANTHROPIC_API_KEY runs the Claude Code CLI and can take
     # OAuth/subscription auth natively (claude-agent-acp, omnigent claude-*).
@@ -558,7 +579,8 @@ def check_subscription_auth(agent: str, required_key: str) -> bool:
     sa = agent_cfg.subscription_auth
     if sa.replaces_env != required_key:
         return False
-    return Path(sa.detect_file).expanduser().is_file()
+    detect_files = sa.detect_files or [sa.detect_file]
+    return any(Path(path).expanduser().is_file() for path in detect_files)
 
 
 def validate_aws_bedrock_env(agent_env: dict[str, str], model: str) -> None: