Skip to content

[BUG] --trials is silently ignored without --matrix #1060

Description

@Benjamin-eecs

Required prerequisites

  • I have searched the Issue Tracker and this has not been reported before.
  • I have verified the problem against the latest main (ff10b7d).

What version of BenchFlow are you using?

0.7.6.dev2102 (PyPI); code paths verified identical on main at ff10b7d.

System information

Host: macOS 15 (arm64), Python 3.12.13. Sandbox: Daytona and Docker (behavior is sandbox-independent).

Problem description

bench eval run --trials N is only consumed by the --matrix expansion. A plain run accepts the flag, silently ignores it, and executes one trial per task. The help text says "Number of trials for --matrix" (src/benchflow/cli/main.py:615), but nothing stops or warns the non-matrix caller, so a batch believed to be N trials produces N=1 data.

We ran a full 16-task x 3-model batch with --trials 3 and got exactly one trial per (task, model) cell before noticing.

Reproducible example code

bench eval run --tasks-dir <dir> --agent oracle --sandbox docker --trials 3

Traceback

No error; exit is clean and the jobs dir contains one rollout per task.

Expected behavior

--trials > 1 without --matrix fails fast with a clean EvalPlanError, the same pattern as the neighboring flag-dependency guards (--registry requires --dataset, --matrix currently requires --tasks-dir).

Additional context

Fix proposed in #1064.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions