Redesign VeRO as a generic program optimization harness

.gitignore

@@ -1,5 +1,6 @@
 # Python-generated files
 __pycache__/
+.pytest_cache/
 *.py[oc]
 build/
 dist/
@@ -9,3 +10,8 @@ wheels/
 # Virtual environments
 .venv
 .env*
+
+# Run secrets loaded via `vero harbor run --env-file` (keep only *.example)
+secrets.env
+*.secrets.env
+!*.example

README.md

@@ -1,122 +1,85 @@
-# VeRO: Versioning Rewards and Observations
+# VeRO: a harness for agents to optimize programs, text, and agents
 
 [![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-
-VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
-
-> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480)
-
-## Repository Structure
-
-```
-vero/
-├── vero/               # Core library (scale-vero)
-├── vero-agents/        # Agent implementations (benchmarking targets)
-├── vero-benchmarking/  # Benchmarking scripts and analysis
-└── LICENSE
-```
-
-### [vero/](vero/)
-
-The core optimization framework. Provides:
-
-- **Policy** — orchestrates the optimization loop (agent + evaluator + git)
-- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK)
-- **Evaluator** — runs task evaluations in isolated subprocess environments
-- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.)
-- **Traces** — session analysis and LLM-based trace interpretation
-
-```bash
-cd vero && uv sync --extra optimize
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
+
+VeRO gives a coding agent something to edit, an evaluation boundary, and durable
+memory of every candidate it tried. The target is anything you can put under Git
+and score — a **program** (a single function up to a whole codebase), **text**
+(a prompt, spec, or config), or an **agent** (its scaffold, tools, and prompts).
+Agents are programs, but not everyone reads "program" that way, so VeRO names
+them explicitly: it was introduced to optimize agents and generalizes the same
+version / evaluate / select loop to any Git-versioned artifact.
+
+```mermaid
+flowchart LR
+    S["Strategy<br/>proposes ideas"] --> P["Candidate producers<br/>edit isolated workspaces"]
+    P --> E["Evaluation backends<br/>score versioned reports"]
+    E --> Sel["Selection keeps the<br/>best feasible candidate"]
+    Sel -->|"next round"| S
 ```
 
-See [vero/README.md](vero/README.md) for full documentation.
-
-### [vero-agents/](vero-agents/)
+The target and evaluator do not need to be Python. External evaluators and
+candidate producers connect through command protocols; Python benchmarks can
+use the optional, optimizer-independent `scale-vero-tasks` package.
 
-Agent implementations used as optimization targets:
+Targets may live locally or in an isolated sandbox. VeRO keeps optimization
+state and experiment tracking on the host while running Git worktrees, producer
+commands, builds, and evaluation commands in the target sandbox. The core guide
+includes a no-bind-mount `DockerSandbox` example.
 
-| Agent | Description |
-|-------|-------------|
-| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. |
-| **web_search_agent** | Web search agent for SimpleQA, Facts Search |
-| **KIRA** | Terminal task agent for Terminal Bench 2.0 |
-| **tau-bench** | Customer service tool-use agent |
-| **pharma_summarizer** | Document summarization agent |
+## Repository layout
 
-See [vero-agents/README.md](vero-agents/README.md) for details.
+| Directory | Purpose |
+| --- | --- |
+| [`vero/`](vero/) | The `scale-vero` optimization kernel, runtime, CLI, and coding-agent adapters |
+| [`vero-tasks/`](vero-tasks/) | Narrow Python task types and schema-v1 evaluation runner |
+| [`program-opt-bench/`](program-opt-bench/) | Harbor-native target programs and end-to-end optimization benchmarks |
 
-### [vero-benchmarking/](vero-benchmarking/)
+Start with the [generic C matrix-multiplication quickstart](vero/examples/c-matmul/),
+try the [26-circle packing benchmark](vero/examples/circle-packing/), or read the
+[core guide](vero/README.md). The C example demonstrates the language-neutral
+command protocol without model credentials. Circle packing is a substantive
+coding-agent benchmark with exact geometry checks and inspectable search
+artifacts.
 
-Scripts and infrastructure for running optimization experiments:
+For a full agent-optimization example, the [GAIA benchmark](program-opt-bench/gaia/)
+pairs a tool-using GPT-5.4 mini target with Harbor's canonical GAIA verifier and
+an immutable 20% / 40% / 40% development, validation, and test split.
 
 ```bash
-cd vero-benchmarking && uv sync --all-extras
-
-# Run an optimization experiment
-uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
-
-# Build datasets
-./scripts/build_datasets.sh
+cd vero
+uv sync --all-extras
+uv run vero --help
 ```
 
-See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation.
-
-## Quick Start
+## Paper reproduction
 
-### Prerequisites
+VeRO was introduced in [*VeRO: A Harness for Agents to Optimize
+Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The current
+library generalizes that version/evaluate/select loop from agents to programs.
 
-- Python 3.11+
-- [uv](https://docs.astral.sh/uv/getting-started/installation/)
-- Git
-- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.)
-
-### Install
+The exact paper implementation is frozen separately from the v0.5 redesign:
 
 ```bash
-git clone <repo-url> && cd vero
-
-# Install core library
-cd vero && uv sync --extra optimize
-
-# Install benchmarking tools
-cd ../vero-benchmarking && uv sync --all-extras
+git checkout paper-v1
 ```
 
-### Run Your First Optimization
-
-```python
-from agents import Agent as OAIAgent
-from vero.policy import Policy
-from vero.agents.vero import VeroAgent
-
-policy = Policy(
-    project_path="/path/to/my-agent",
-    dataset="/path/to/my-dataset",
-    agent=VeroAgent(
-        oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"),
-    ),
-    task="main",
-    train_budget=10,
-    max_turns=200,
-)
-
-best = await policy.run()
-print(f"Best commit: {best.commit}, score: {best.score}")
-```
+Use the `paper/v1` branch or `paper-v1` tag for reproduction. Development of
+the generic program optimizer continues on the `v0.5` branch. The frozen ref
+also preserves the paper-era `vero-agents` and `vero-benchmarking` directories;
+their Harbor-native replacement on `v0.5` is `program-opt-bench`.
 
 ## Citation
 
 ```bibtex
 @article{ursekar2026vero,
-  title={VeRO: An Evaluation Harness for Agents to Optimize Agents},
+  title={VeRO: A Harness for Agents to Optimize Agents},
   author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam},
   journal={arXiv preprint arXiv:2602.22480},
   year={2026}
 }
 ```
 
-## License
-
-[MIT](LICENSE)
+VeRO is licensed under the [MIT License](LICENSE).

program-opt-bench/README.md

@@ -0,0 +1,29 @@
+# Program optimization benchmarks
+
+`program-opt-bench` contains end-to-end benchmarks for improving programs with
+VeRO. Each leaf directory pairs one editable target program with one immutable
+Harbor dataset and compiles them into an outer Harbor optimization task.
+
+The benchmark definitions intentionally keep three boundaries visible:
+
+- `target/` is the program the optimization agent may edit.
+- `partitions/` pins the cases and the development/validation/test split.
+- `build.yaml` is trusted configuration: model, evaluator, access policy,
+  budgets, and final scoring.
+
+In each benchmark, the complete development tasks and attachments are mounted
+read-only for the optimization agent. Development evaluations expose per-case
+results and complete Harbor trial records, including exact failures and
+target-agent logs. Validation remains aggregate-only, and
+test is reachable only by the trusted final verifier.
+
+The paper-era benchmark stack remains available on the `paper/v1` branch and
+the `paper-v1` tag. New benchmarks should use this Harbor-native layout.
+
+## Benchmarks
+
+| Benchmark | Editable target | Dataset | Split |
+| --- | --- | --- | --- |
+| [GAIA baseline](gaia/baseline/) | Tool-using Responses API agent | Harbor `gaia/gaia` | 20% / 40% / 40% |
+| [SWE-Atlas-QnA baseline](swe-atlas-qna/baseline/) | Codebase investigation agent | Harbor `scale-ai/swe-atlas-qna` | 20% / 40% / 40% |
+| [tau3 baseline](tau3/baseline/) | MCP customer-service agent | Harbor `sierra-research/tau3-bench` | 20% / 40% / 40% |

program-opt-bench/ale-bench/.gitignore

@@ -0,0 +1,2 @@
+environment/vero/
+runs/

program-opt-bench/ale-bench/environment/Dockerfile

@@ -0,0 +1,19 @@
+# Main service: the optimizer workbench a coding agent works in.
+# Holds the editable target (solution.cpp) plus the VeRO CLI so the agent can
+# call `vero harbor eval` / `vero harbor status` against the trusted sidecar.
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# VeRO is not on public PyPI; vendor it into the build context first
+# (`cp -r <repo>/vero environment/vero`) then install it.
+COPY vero /opt/vero
+RUN uv pip install --system "/opt/vero[harbor]"
+
+COPY agent-seed /opt/agent-seed
+COPY main/seed.sh /opt/seed.sh
+RUN chmod +x /opt/seed.sh && useradd -m -u 1001 agent
+
+WORKDIR /work/agent

program-opt-bench/ale-bench/environment/agent-baseline/solution.cpp

@@ -0,0 +1,8 @@
+#include <bits/stdc++.h>
+int main(){
+    std::ios::sync_with_stdio(false);
+    std::string line;
+    while (std::getline(std::cin, line)) {}
+    std::cout << "\n";
+    return 0;
+}

program-opt-bench/ale-bench/environment/agent-seed/solution.cpp

@@ -0,0 +1,8 @@
+#include <bits/stdc++.h>
+int main(){
+    std::ios::sync_with_stdio(false);
+    std::string line;
+    while (std::getline(std::cin, line)) {}
+    std::cout << "\n";
+    return 0;
+}

program-opt-bench/ale-bench/environment/docker-compose.yaml

@@ -0,0 +1,60 @@
+# Harbor merges this after its generated main-service configuration.
+services:
+  main:
+    command: ["/opt/seed.sh"]
+    environment:
+      VERO_EVAL_URL: "http://eval-sidecar:8000"
+      OPENAI_API_KEY: "${OPENAI_API_KEY:?OPENAI_API_KEY must be set}"
+      OPENAI_BASE_URL: "${OPENAI_BASE_URL:?OPENAI_BASE_URL must be set}"
+    volumes:
+      - agent_repo:/work/agent
+      - token_state:/state/token:ro
+    depends_on:
+      eval-sidecar:
+        condition: service_healthy
+
+  eval-sidecar:
+    build:
+      context: .
+      dockerfile: sidecar/Dockerfile
+    # Runs its own Docker daemon (DinD) so ALE-Bench's sibling judge/tool
+    # containers share this container's filesystem — required for their bind
+    # mounts to resolve. Needs privileged + host cgroup ns (nested cgroup v2
+    # otherwise fails: "cannot enter cgroupv2 ... it is in threaded mode").
+    privileged: true
+    cgroup: host
+    command:
+      - "vero"
+      - "harbor"
+      - "serve"
+      - "--factory"
+      - "ale_factory:build"
+      - "--config"
+      - "/opt/serve.json"
+      - "--admin-token"
+      - "/state/token/admin.token"
+    environment:
+      # ALE-Bench talks to the in-container dockerd started by the entrypoint.
+      DOCKER_HOST: "unix:///var/run/docker.sock"
+      ALE_BENCH_DATA: "/state/ale-data"
+    volumes:
+      - agent_repo:/work/agent:ro
+      - admin_state:/state/admin
+      - token_state:/state/token
+      - ale_data:/state/ale-data
+      # DinD storage (avoids overlay-on-overlay issues for the inner daemon).
+      - docker_storage:/var/lib/docker
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
+      interval: 5s
+      timeout: 10s
+      # The entrypoint starts dockerd + pulls the ~4GB judge image before serving.
+      retries: 40
+      start_period: 180s
+
+volumes:
+  agent_repo:
+  admin_state:
+  token_state:
+  ale_data:
+  docker_storage:

program-opt-bench/ale-bench/environment/main/seed.sh

@@ -0,0 +1,12 @@
+#!/bin/sh
+set -eu
+if [ ! -d /work/agent/.git ]; then
+  cp -a /opt/agent-seed/. /work/agent/
+  cd /work/agent
+  git init -q
+  git add -A
+  git -c user.email=seed@vero.test -c user.name=seed commit -qm "baseline"
+fi
+find /work/agent -exec chown agent:agent {} +
+git config --system --add safe.directory /work/agent
+exec sleep infinity

program-opt-bench/ale-bench/environment/sidecar/Dockerfile

@@ -0,0 +1,42 @@
+# Trusted evaluation sidecar. It owns scoring, budget, disclosure, and final
+# selection. The inner loop is a plain CommandBackend (harness/score.py), which
+# calls ALE-Bench's deterministic judge. ALE-Bench spawns the C++ judge + its
+# Rust-tool builder as sibling Docker containers with bind mounts, so this
+# container runs its OWN Docker daemon (DinD) — the sibling containers must share
+# this container's filesystem for their mounts to resolve. Requires privileged.
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        git ca-certificates curl \
+        docker.io iptables \
+        libcairo2 libffi8 \
+    && rm -rf /var/lib/apt/lists/*
+
+# VeRO (not on public PyPI): vendor into the build context, then install.
+COPY vero /opt/vero
+RUN uv pip install --system "/opt/vero[harbor]"
+
+# ALE-Bench toolkit (the judge orchestration + scoring).
+RUN uv pip install --system "git+https://github.com/SakanaAI/ALE-Bench.git"
+
+COPY agent-baseline /opt/agent-baseline
+COPY sidecar/ale_factory.py /opt/ale_factory.py
+COPY sidecar/harness /opt/harness
+COPY sidecar/serve.json /opt/serve.json
+COPY sidecar/dind-entrypoint.sh /opt/dind-entrypoint.sh
+RUN chmod +x /opt/dind-entrypoint.sh
+
+RUN cd /opt/agent-baseline \
+    && git init -q \
+    && git add -A \
+    && git -c user.email=baseline@vero.test -c user.name=baseline commit -qm "baseline" \
+    && git config --system --add safe.directory /opt/agent-baseline \
+    && git config --system --add safe.directory /work/agent \
+    && git config --system --add safe.directory /work/agent/.git
+
+ENV PYTHONPATH=/opt
+WORKDIR /opt
+# Start the in-container Docker daemon + bootstrap the judge image, then exec the
+# serve command passed by docker-compose.
+ENTRYPOINT ["/opt/dind-entrypoint.sh"]