Skip to content

Add RepoTrials to eval frameworks and harnesses - #78

Open
PozziTiv4ik wants to merge 1 commit into
benchflow-ai:mainfrom
PozziTiv4ik:add-repotrials
Open

Add RepoTrials to eval frameworks and harnesses#78
PozziTiv4ik wants to merge 1 commit into
benchflow-ai:mainfrom
PozziTiv4ik:add-repotrials

Conversation

@PozziTiv4ik

Copy link
Copy Markdown

What changed

Adds RepoTrials to §5a · Eval frameworks & harnesses (code-first test-runners).

Why it belongs

RepoTrials is an Apache-2.0, local-first Python CLI for building repository-specific coding-agent evaluations from real fixes already present in Git history. It reconstructs the pre-fix state, seals task material, validates BASE/RED/GOLD test transitions, runs repeated agent trials, and grades behavior with hidden tests rather than gold-diff similarity. It can also export tasks for Harbor.

The specific gap it addresses is task creation for a team's own repository: instead of relying only on a fixed public corpus or requiring every task to be authored by hand, it turns reviewed historical fixes into reproducible private evaluations.

Disclosure and maturity

I am submitting this on behalf of the project's maintainer and owner, @PozziTiv4ik. Version v0.1.0 is the first public release. The repository has a working demo and automated tests, but I am not claiming independent adoption or production maturity.

Validation

  • Changed only README.md, with one added entry.
  • git diff --check passes.
  • Verified the canonical repository URL returns HTTP 200.
  • Checked the current open pull requests for an existing RepoTrials submission; none was present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant