Required prerequisites
What version of BenchFlow are you using?
0.7.6.dev2102 (PyPI); code paths verified identical on main at ff10b7d.
System information
Host: macOS 15 (arm64), Python 3.12.13. Sandbox: Daytona and Docker (behavior is sandbox-independent).
Problem description
bench eval run --trials N is only consumed by the --matrix expansion. A plain run accepts the flag, silently ignores it, and executes one trial per task. The help text says "Number of trials for --matrix" (src/benchflow/cli/main.py:615), but nothing stops or warns the non-matrix caller, so a batch believed to be N trials produces N=1 data.
We ran a full 16-task x 3-model batch with --trials 3 and got exactly one trial per (task, model) cell before noticing.
Reproducible example code
bench eval run --tasks-dir <dir> --agent oracle --sandbox docker --trials 3
Traceback
No error; exit is clean and the jobs dir contains one rollout per task.
Expected behavior
--trials > 1 without --matrix fails fast with a clean EvalPlanError, the same pattern as the neighboring flag-dependency guards (--registry requires --dataset, --matrix currently requires --tasks-dir).
Additional context
Fix proposed in #1064.
Required prerequisites
main(ff10b7d).What version of BenchFlow are you using?
0.7.6.dev2102 (PyPI); code paths verified identical on
mainatff10b7d.System information
Host: macOS 15 (arm64), Python 3.12.13. Sandbox: Daytona and Docker (behavior is sandbox-independent).
Problem description
bench eval run --trials Nis only consumed by the--matrixexpansion. A plain run accepts the flag, silently ignores it, and executes one trial per task. The help text says "Number of trials for --matrix" (src/benchflow/cli/main.py:615), but nothing stops or warns the non-matrix caller, so a batch believed to be N trials produces N=1 data.We ran a full 16-task x 3-model batch with
--trials 3and got exactly one trial per (task, model) cell before noticing.Reproducible example code
Traceback
No error; exit is clean and the jobs dir contains one rollout per task.
Expected behavior
--trials > 1without--matrixfails fast with a cleanEvalPlanError, the same pattern as the neighboring flag-dependency guards (--registry requires --dataset,--matrix currently requires --tasks-dir).Additional context
Fix proposed in #1064.