Redesign VeRO as a generic program optimization harness

[varunursekar]

Summary

• replace the paper-era Policy and experiment model with backend-neutral candidate, evaluation, optimization, runtime, workspace, and coding-agent interfaces
• support language-neutral command evaluators, optimizer-independent Python tasks, and Harbor as a canonical evaluation backend
• add a declarative vero.toml workflow, baseline-only vero evaluate, repeatable vero run, and a complete CLI for objectives, constraints, case selection, refs, environments, limits, and concurrency
• add a checked-in C matrix-multiplication quickstart that compiles, checks correctness, benchmarks, optimizes, selects the faster feasible commit, preserves the target checkout, and cleans up worktrees
• support sequential and parallel candidate production, durable budgets and reports, external producers, built-in Vero and Claude coding agents, and cancellation-safe concurrent batches
• make session resume restore candidate lineage, completed rounds, compatible evaluation history, and supported coding-agent state
• add recursive session discovery, aggregate evaluation inspection, and optional W&B reporting over canonical runtime events
• add Harbor sidecar, verifier, transport, HTTP/CLI, and task compiler support with trusted policy and finalization boundaries
• retain both Python and non-Python matrix-multiplication examples while keeping exact paper reproduction on paper/v1 and the paper-v1 tag

Harbor and Modal

• support Harbor at the outer level by compiling a VeRO optimization into a canonical Harbor task
• support Harbor at the inner level by evaluating candidate programs through nested harbor run jobs
• propagate required task-environment secrets to remote Harbor environments without embedding their values in the compiled task
• support pinned Harbor environment extras such as harbor[modal]==0.18.0 and select an explicit inner Python version
• preserve both local Harbor selectors and canonical result task names
• import candidates safely across the outer agent/sidecar UID boundary using exact trusted Git directories

Hardening

• terminate complete host process groups on timeout/cancellation and terminate the in-container process group for Docker executions
• reconcile database.json from canonical completed evaluation directories on resume, closing the write-order crash window
• persist cancelled evaluations as terminal records and make durable budget reservation disk/memory atomic under cancellation; budgets explicitly meter attempted runs
• use detached candidate worktrees, roll back interrupted copies, and retain evaluated commits under durable hidden refs/vero/... refs without leaking normal branches
• make the generic evaluation engine fail closed unless authorization is configured; the trusted local factory opts into an explicit allow-all resolver
• validate optimizer configuration as a discriminated command/agent union so irrelevant fields cannot be silently ignored
• map backend request rejection to HTTP 400, separate producer and evaluation timeouts in the CLI, lock database readers, and move remaining durable writes off async event loops
• document aggregate-differencing and shared-container admin-token trust assumptions

Validation

• complete suite: 379 passed, 6 skipped
• Ruff checks pass for all changed Python files
• source distribution and wheel build successfully
• checked-in C optimizer and advertised Python evaluation example pass end to end
• live credentialed VeroAgent C matrix-multiplication optimization completed with five agent checkpoints plus baseline and selected an improved candidate (about 5.5x faster in that run)
• both tracked uv lockfiles pass uv lock --check
• isolated wheel imports pass for base, Claude, optimize, and Harbor dependency sets
• deterministic outer-Harbor-on-Modal smoke completed with reward 1.0 and no exceptions
• real outer Codex (gpt-5) run completed the full nested workflow on Modal: status, candidate validation through inner Harbor, submission, trusted hidden final evaluation through inner Harbor, and outer reward 1.0 with no exceptions

The final live topology is:

Harbor (Modal, Codex) -> VeRO sidecar -> Harbor validation (Modal) -> submit -> VeRO verifier -> Harbor hidden final (Modal)

Greptile Summary

This PR replaces the paper-era VeRO implementation with a fully generic program optimization harness: backend-neutral candidate/evaluation/optimization/runtime interfaces, a declarative vero.toml workflow, Harbor/Modal integration as both outer task and inner evaluation backend, and hardened worktree, budget, and session lifecycle management.

• Evaluation engine: new EvaluationEngine with authorization resolver, discriminated-union optimizer config, asyncio.to_thread for all durable writes (engine, budget, sidecar, session), and asyncio.TaskGroup-based parallel candidate production replacing the previous asyncio.gather approach.
• Harbor integration: GitCandidateTransport fetches via unique temporary refs to avoid FETCH_HEAD races, retains commits under hidden refs/vero/… refs, and validates all paths and symlinks before extraction in the build compiler.
• Unshielded cleanup: in Optimizer._produce_candidate, the finally: await candidate_workspace.destroy() is not wrapped with asyncio.shield, so when TaskGroup cancels sibling tasks the git worktree removal can be interrupted mid-execution, leaving worktrees registered and on disk — unlike temp_copy, which already uses asyncio.shield for the same cleanup.

Confidence Score: 4/5

Safe to merge with the worktree cleanup fix; all other hardening, atomicity, and security changes look correct.

The entire new evaluation/optimization/harbor stack is well-structured and the previously-flagged blocking-write issues are resolved with asyncio.to_thread throughout. One gap remains: Optimizer._produce_candidate calls await candidate_workspace.destroy() in a bare finally block that is not shielded, so a TaskGroup cancellation during parallel production can abort the git worktree teardown mid-way, leaving worktrees registered on disk. All other cleanup paths (temp_copy, budget reservation, session manifest) are already shielded consistently.

vero/src/vero/optimization/optimizer.py — the _produce_candidate finally block needs asyncio.shield on destroy().

Important Files Changed

Filename	Overview
vero/src/vero/optimization/optimizer.py	New batch-oriented optimizer with parallel candidate production; unshielded destroy() in finally block can leave git worktrees allocated when a parallel task is cancelled by TaskGroup.
vero/src/vero/harbor/transport.py	New GitCandidateTransport: resolves agent ref to an object ID before fetching, uses a unique temporary ref to avoid FETCH_HEAD races, retains commits under refs/vero/candidates/, and validates all paths as absolute.
vero/src/vero/evaluation/engine.py	New EvaluationEngine with authorization resolver, budget metering, and atomic asyncio.to_thread persistence. Correctly wraps the synchronous write in to_thread (previously-flagged issue is addressed).
vero/src/vero/harbor/verifier.py	New CanonicalVerifier for final candidate scoring; the blocking _atomic_write_json inside the asyncio.Lock in finalize was flagged in a previous thread and remains unaddressed.
vero/src/vero/workspace/git.py	Refactored worktree management to use --detach, roll back interrupted copies, propagate project_path correctly into copies; temp_copy correctly shields cleanup with asyncio.shield.
vero/src/vero/harbor/sidecar.py	Agent-facing sidecar with access policies, aggregate floor enforcement, and correct asyncio.to_thread offloading; previously-flagged sync-write issue is resolved.
vero/src/vero/evaluation/security.py	New sanitize_evaluation_report helper that redacts secrets from all free-form report fields while preserving structural identifiers.
vero/src/vero/harbor/build/compiler.py	New Harbor task compiler; _safe_extract_tar validates all member paths and symlink targets before extraction.
vero/src/vero/config.py	New declarative vero.toml config system with discriminated-union OptimizerConfig validation and path resolution beside the config file.
vero/src/vero/runtime/session.py	New durable OptimizationSession with manifest reconciliation on resume and asyncio.to_thread for all manifest writes.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant CLI as vero CLI
    participant Session as OptimizationSession
    participant Optimizer as Optimizer
    participant TG as asyncio.TaskGroup
    participant Producer as CandidateProducer
    participant Workspace as GitWorkspace copy
    participant Engine as EvaluationEngine
    participant Backend as EvaluationBackend

    CLI->>Session: run()
    Session->>Optimizer: run(baseline)
    Optimizer->>Engine: evaluate_candidate(baseline)
    Engine->>Backend: evaluate(request)
    Backend-->>Engine: EvaluationReport
    Engine-->>Optimizer: EvaluationRecord baseline

    loop max_rounds
        Optimizer->>Optimizer: strategy.propose(context)
        Optimizer->>TG: create tasks parallel proposals
        TG->>Workspace: copy from_version parent.version
        TG->>Producer: produce workspace evaluation_gateway
        Producer->>Engine: evaluate_current optional checkpoints
        Engine-->>Producer: EvaluationRecord
        Producer-->>TG: CandidateChange
        Note over TG,Workspace: finally await destroy not shielded
        TG->>Engine: evaluate_candidate new_candidate
        Engine->>Backend: evaluate request
        Backend-->>Engine: EvaluationReport
        Engine-->>Optimizer: EvaluationRecord
    end

    Optimizer-->>Session: OptimizationResult
    Session-->>CLI: best candidate

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant CLI as vero CLI
    participant Session as OptimizationSession
    participant Optimizer as Optimizer
    participant TG as asyncio.TaskGroup
    participant Producer as CandidateProducer
    participant Workspace as GitWorkspace copy
    participant Engine as EvaluationEngine
    participant Backend as EvaluationBackend

    CLI->>Session: run()
    Session->>Optimizer: run(baseline)
    Optimizer->>Engine: evaluate_candidate(baseline)
    Engine->>Backend: evaluate(request)
    Backend-->>Engine: EvaluationReport
    Engine-->>Optimizer: EvaluationRecord baseline

    loop max_rounds
        Optimizer->>Optimizer: strategy.propose(context)
        Optimizer->>TG: create tasks parallel proposals
        TG->>Workspace: copy from_version parent.version
        TG->>Producer: produce workspace evaluation_gateway
        Producer->>Engine: evaluate_current optional checkpoints
        Engine-->>Producer: EvaluationRecord
        Producer-->>TG: CandidateChange
        Note over TG,Workspace: finally await destroy not shielded
        TG->>Engine: evaluate_candidate new_candidate
        Engine->>Backend: evaluate request
        Backend-->>Engine: EvaluationReport
        Engine-->>Optimizer: EvaluationRecord
    end

    Optimizer-->>Session: OptimizationResult
    Session-->>CLI: best candidate

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
vero/src/vero/optimization/optimizer.py:281-282
**Unshielded worktree cleanup under `TaskGroup` cancellation**

`await candidate_workspace.destroy()` in the `finally` block is not wrapped with `asyncio.shield`. When `max_concurrency > 1` and one `produce(proposal)` task raises, `asyncio.TaskGroup` cancels all remaining tasks by calling `task.cancel()`. Python raises `CancelledError` at the first `await` the cancelled task encounters — which is the `await candidate_workspace.destroy()` call here. The `destroy()` coroutine is then aborted at its own internal `await` (e.g., the `self._sandbox.run` call inside `_remove_worktree`), leaving the git worktree registered and on disk. The `temp_copy` context manager already uses `asyncio.shield` for the same reason, so the fix is consistent with existing practice.

_{Reviews (6): Last reviewed commit: "Document nested Modal Harbor workflow" | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​orjson@​3.11.5 ⏵ 3.11.9	+1	+16			-30
	pypi/​wandb@​0.23.1 ⏵ 0.28.0
	pypi/​harbor@​0.20.0 ⏵ 0.18.0
	pypi/​click@​8.3.1 ⏵ 8.4.2	+1
	pypi/​uvicorn@​0.44.0 ⏵ 0.51.0	+1
	pypi/​pydantic@​2.13.4
	pypi/​fastapi@​0.139.0
	pypi/​fastapi@​0.139.2

View full report

[socket-security]

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

[varunursekar]

Pushed a batch of work onto v0.5 (clean fast-forward). Highlights since the last state:

tau3 benchmark now runs end-to-end

• Collapsed the tau3 target to a single host-side Tau3Agent (deleted runner.py + the os.environ cred-forward); it drives the container-internal MCP server via exec-curl and makes target-model calls host-side.
• task_services_use_upstream: routes a task's own LLM eval services (tau3's user-simulator + NL-assertions grader, which run in the task containers and can't reach the compose-internal gateway) to the real upstream via the trusted eval-sidecar, while the candidate agent keeps the metered, model-allow-listed gateway (VERO_AGENT_INFERENCE_*). Optimizer/main never sees the raw upstream. Verified live: real graded rewards, no connection/credential errors.

Budget-blindness

• disclose_budget toggle: hides all budget signal from the optimizer (instruction + vero harbor status) while enforcement is unchanged — enables budget-blind optimization runs to test whether budget-awareness is what stops quality from scaling with budget.

Strategy

• DarwinGodelStrategy: open-ended archive evolution (whole-archive parent sampling ∝ score/(1+children)).

Full vero test suite green (327 passed, 5 skipped).