Add SWE-bench Pro eval preflight and harness docs

README.md

@@ -136,6 +136,18 @@ This will create a JSON file in the format expected by the evaluation script:
 ]
 ```
 
+### 2b. Preflight (recommended)
+
+Before evaluation, validate patch JSON and `run_scripts/` layout:
+
+```bash
+python scripts/swebench_preflight.py \
+  --patches sample1_patches.json \
+  --scripts-dir run_scripts
+```
+
+See [docs/HARNESS.md](docs/HARNESS.md) and [docs/GOLDEN_PATCH.md](docs/GOLDEN_PATCH.md) for harness invariants and golden-patch verification ([#101](https://github.com/scaleapi/SWE-bench_Pro-os/issues/101)).
+
 ### 3. Evaluate Patches
 
 Evaluate patch predictions on SWE-Bench Pro:

docs/GOLDEN_PATCH.md

@@ -0,0 +1,63 @@
+# Golden patch verification
+
+Issue [#101](https://github.com/scaleapi/SWE-bench_Pro-os/issues/101): running evaluation with
+dataset golden patches can fail on some instances (e.g. NodeBB) with:
+
+```text
+[emailer.send] Error: [[error:sendmail-not-found]]
+```
+
+The tests in the log may still pass — the error is logged during email setup, not necessarily
+as a test failure. Always inspect `test_output.txt` / parser output, not only stderr noise.
+
+## Recommended workflow
+
+### 1. Extract gold patches
+
+```bash
+python helper_code/extract_gold_patches.py --output gold_patches.json
+```
+
+### 2. Preflight
+
+```bash
+python scripts/swebench_preflight.py \
+  --patches gold_patches.json \
+  --scripts-dir run_scripts
+```
+
+NodeBB instances will emit a **WARN** about sendmail — expected for #101-class images.
+
+### 3. Evaluate one instance (local Docker)
+
+```bash
+python swe_bench_pro_eval.py \
+  --raw_sample_path=<csv_with_one_row> \
+  --patch_path=gold_patches.json \
+  --output_dir=out/golden-verify/ \
+  --scripts_dir=run_scripts \
+  --use_local_docker \
+  --num_workers=1
+```
+
+### 4. Single-instance smoke (example)
+
+Instance from #101:
+
+`instance_NodeBB__NodeBB-04998908ba6721d64eba79ae3b65a351dcfbc5b5-vnan`
+
+Filter `gold_patches.json` and dataset CSV to this `instance_id` before running step 3.
+
+## Interpreting results
+
+| Signal | Meaning |
+|--------|---------|
+| Parser reports fail-to-pass tests passed | Golden patch valid for grading |
+| sendmail error in logs only | Environment noise (#101) — check parser |
+| `git apply` failure | Patch/instance mismatch — fix harness before trusting scores |
+| Missing `run_scripts/` files | Run preflight — do not publish scores |
+
+## Related issues
+
+- [#93](https://github.com/scaleapi/SWE-bench_Pro-os/issues/93) — git history reward hacking
+- [#108](https://github.com/scaleapi/SWE-bench_Pro-os/issues/108) — determinacy audit

docs/HARNESS.md

@@ -0,0 +1,53 @@
+# SWE-bench Pro harness
+
+This document describes the grading harness invariants for SWE-bench Pro OSS evals.
+Run `python scripts/swebench_preflight.py` before `swe_bench_pro_eval.py` to catch
+layout and schema issues early.
+
+## Pipeline overview
+
+1. **Generate patches** — agent harness (SWE-agent, mini-swe-agent, etc.) writes `.pred` files
+2. **Gather patches** — `helper_code/gather_patches.py` → JSON array
+3. **Preflight** — `scripts/swebench_preflight.py` validates JSON + `run_scripts/`
+4. **Evaluate** — `swe_bench_pro_eval.py` applies patches in Docker/Modal and runs tests
+
+## Patch JSON format
+
+```json
+[
+  {
+    "instance_id": "instance_org__repo-<hash>",
+    "patch": "diff --git a/...",
+    "prefix": "gold"
+  }
+]
+```
+
+Required keys: `instance_id`, `patch`, `prefix`. Duplicate `instance_id` values are rejected by preflight.
+
+## Per-instance run scripts
+
+For each `instance_id`, the repo ships:
+
+- `run_scripts/<instance_id>/run_script.sh` — applies patch and runs tests
+- `run_scripts/<instance_id>/parser.py` — parses stdout into pass/fail
+
+Preflight fails if either file is missing for a patch entry.
+
+## Known harness footguns
+
+| Issue | Symptom | Mitigation |
+|-------|---------|------------|
+| [#101](https://github.com/scaleapi/SWE-bench_Pro-os/issues/101) golden patch / sendmail | `[[error:sendmail-not-found]]` in NodeBB email tests | See [GOLDEN_PATCH.md](GOLDEN_PATCH.md) |
+| [#93](https://github.com/scaleapi/SWE-bench_Pro-os/issues/93) git history leakage | Agent can `git show` future fix commits | Strip future history in Docker images before eval |
+| [#6](https://github.com/scaleapi/SWE-bench_Pro-os/issues/6) bash entrypoint | Manual `bash` breaks eval | Use image entrypoint as documented |
+
+## Binary patch hunks
+
+`swe_bench_pro_eval.py` strips binary diff sections before apply. Preflight warns when
+binary hunks are present so maintainers know patches may be incomplete.
+
+## References
+
+- [SWE-bench Pro paper](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20(9).pdf)
+- [HuggingFace dataset](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)

scripts/swebench_preflight.py

@@ -0,0 +1,196 @@
+#!/usr/bin/env python3
+"""Preflight checks before running SWE-bench Pro evaluation.
+
+Validates patch JSON, run_scripts layout, and known harness footguns (#101, #93)
+before invoking swe_bench_pro_eval.py.
+
+Usage:
+  python scripts/swebench_preflight.py --patches gold_patches.json --scripts-dir run_scripts
+  python scripts/swebench_preflight.py --patches preds.json --scripts-dir run_scripts --json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+REQUIRED_PATCH_KEYS = ("instance_id", "patch", "prefix")
+KNOWN_WARNINGS: dict[str, str] = {
+    "NodeBB": (
+        "NodeBB instances may log sendmail-not-found during email tests when running "
+        "golden patches (#101). Tests may still pass; see docs/GOLDEN_PATCH.md."
+    ),
+}
+
+
+def load_patches(path: Path) -> list[dict[str, Any]]:
+    data = json.loads(path.read_text())
+    if not isinstance(data, list):
+        raise ValueError(f"{path}: expected JSON array of patch objects")
+    return data
+
+
+def validate_patches(patches: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], list[str]]:
+    gates: list[dict[str, Any]] = []
+    errors: list[str] = []
+    seen: set[str] = set()
+
+    for i, entry in enumerate(patches):
+        if not isinstance(entry, dict):
+            errors.append(f"entry {i}: expected object")
+            continue
+        missing = [k for k in REQUIRED_PATCH_KEYS if k not in entry]
+        if missing:
+            errors.append(f"entry {i}: missing keys {missing}")
+            continue
+        iid = entry["instance_id"]
+        if iid in seen:
+            errors.append(f"duplicate instance_id: {iid}")
+        seen.add(iid)
+        if not str(entry.get("patch", "")).strip():
+            errors.append(f"{iid}: empty patch")
+
+    gates.append(
+        {
+            "name": "patch_schema",
+            "passed": not errors,
+            "detail": f"{len(patches)} patch(es), {len(seen)} unique instance_id(s)",
+        }
+    )
+    return gates, errors
+
+
+def validate_run_scripts(
+    patches: list[dict[str, Any]], scripts_dir: Path
+) -> tuple[list[dict[str, Any]], list[str]]:
+    gates: list[dict[str, Any]] = []
+    errors: list[str] = []
+    missing_scripts: list[str] = []
+
+    for entry in patches:
+        iid = entry["instance_id"]
+        inst_dir = scripts_dir / iid
+        run_script = inst_dir / "run_script.sh"
+        parser = inst_dir / "parser.py"
+        if not run_script.is_file():
+            missing_scripts.append(iid)
+            errors.append(f"{iid}: missing {run_script}")
+        if not parser.is_file():
+            errors.append(f"{iid}: missing {parser}")
+
+    gates.append(
+        {
+            "name": "run_scripts_layout",
+            "passed": not missing_scripts,
+            "detail": (
+                f"all {len(patches)} instance run_scripts present"
+                if not missing_scripts
+                else f"missing run_script for {len(missing_scripts)} instance(s)"
+            ),
+        }
+    )
+    return gates, errors
+
+
+def collect_known_warnings(patches: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    warnings: list[dict[str, Any]] = []
+    for entry in patches:
+        iid = entry["instance_id"]
+        for needle, message in KNOWN_WARNINGS.items():
+            if needle in iid:
+                warnings.append({"instance_id": iid, "warning": message})
+    return warnings
+
+
+def check_binary_patch_sections(patches: list[dict[str, Any]]) -> dict[str, Any]:
+    binary_hits = 0
+    for entry in patches:
+        patch = entry.get("patch", "")
+        if re.search(r"^Binary files .* differ$", patch, re.MULTILINE):
+            binary_hits += 1
+    return {
+        "name": "binary_patch_sections",
+        "passed": True,
+        "detail": (
+            f"{binary_hits} patch(es) contain binary hunks (stripped at eval time)"
+            if binary_hits
+            else "no binary hunks detected"
+        ),
+        "warn": binary_hits > 0,
+    }
+
+
+def run_preflight(patches_path: Path, scripts_dir: Path) -> dict[str, Any]:
+    patches = load_patches(patches_path)
+    gates: list[dict[str, Any]] = []
+    all_errors: list[str] = []
+
+    g1, e1 = validate_patches(patches)
+    gates.extend(g1)
+    all_errors.extend(e1)
+
+    g2, e2 = validate_run_scripts(patches, scripts_dir)
+    gates.extend(g2)
+    all_errors.extend(e2)
+
+    gates.append(check_binary_patch_sections(patches))
+    warnings = collect_known_warnings(patches)
+
+    active = [g for g in gates if not g.get("warn")]
+    passed = all(g["passed"] for g in active)
+
+    return {
+        "patches": str(patches_path.resolve()),
+        "scripts_dir": str(scripts_dir.resolve()),
+        "patch_count": len(patches),
+        "gates": gates,
+        "warnings": warnings,
+        "errors": all_errors,
+        "passed": passed,
+    }
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description="SWE-bench Pro eval preflight")
+    parser.add_argument("--patches", type=Path, required=True, help="Patch JSON for eval")
+    parser.add_argument(
+        "--scripts-dir",
+        type=Path,
+        default=Path("run_scripts"),
+        help="Directory with per-instance run_script.sh (default: run_scripts)",
+    )
+    parser.add_argument("--json", action="store_true", help="Emit machine-readable report")
+    args = parser.parse_args(argv)
+
+    if not args.patches.is_file():
+        print(f"error: patches file not found: {args.patches}", file=sys.stderr)
+        return 1
+    if not args.scripts_dir.is_dir():
+        print(f"error: scripts dir not found: {args.scripts_dir}", file=sys.stderr)
+        return 1
+
+    report = run_preflight(args.patches, args.scripts_dir)
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        status = "PASS" if report["passed"] else "FAIL"
+        print(f"SWE-bench Pro preflight — {status}")
+        for gate in report["gates"]:
+            mark = "PASS" if gate["passed"] else "FAIL"
+            if gate.get("warn"):
+                mark = "WARN"
+            print(f"  [{mark}] {gate['name']}: {gate['detail']}")
+        for w in report["warnings"]:
+            print(f"  [WARN] {w['instance_id']}: {w['warning']}")
+        for err in report["errors"]:
+            print(f"  [ERR] {err}")
+
+    return 0 if report["passed"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())