Add SWE-bench Pro eval preflight and harness docs

[Abhishek21g]

Problem

Teams running golden-patch or agent evals hit silent harness failures — missing run_scripts/, duplicate instance_ids, and known env footguns like NodeBB sendmail errors (#101).

Fix

• scripts/swebench_preflight.py — validate patch JSON + per-instance run_scripts/ before swe_bench_pro_eval.py
• docs/HARNESS.md — harness invariants and footgun table (Golden patch failed #101, Git Reward Hacking in SWEBench Pro OSS #93, Issues running the images #6)
• docs/GOLDEN_PATCH.md — golden-patch verification workflow for Golden patch failed #101
• tests/test_swebench_preflight.py — unittest coverage

Verification

python -m unittest tests.test_swebench_preflight -v
python scripts/swebench_preflight.py --patches gold_patches.json --scripts-dir run_scripts

Closes #101 (docs/workflow); does not change Docker images.

Greptile Summary

This PR introduces a preflight validation script (scripts/swebench_preflight.py) that catches common harness failures — missing run_scripts/, duplicate instance_ids, binary hunks, and known NodeBB footguns — before swe_bench_pro_eval.py is invoked, along with accompanying docs and unit tests.

• swebench_preflight.py runs four gates (patch schema, run-scripts layout, binary hunks as warn-only, known warnings) and exits non-zero on any hard failure; all gate functions correctly guard against non-dict and missing-key entries.
• docs/HARNESS.md and docs/GOLDEN_PATCH.md document harness invariants and the golden-patch verification workflow for issues Golden patch failed #101/Git Reward Hacking in SWEBench Pro OSS #93.
• The test suite covers all major failure paths, though temp dirs created by _layout() are never cleaned up, and main() is missing a try/except around the load_patches call so malformed/non-array JSON files produce a raw traceback instead of the clean error: … message the rest of main() uses.

Confidence Score: 5/5

Safe to merge — all changes are additive and do not touch any existing evaluation code or Docker images.

The gate logic is correct and well-covered by tests. The two minor gaps (uncaught exception in main() on malformed JSON input, and temp-dir leak in tests) do not affect preflight correctness or any production path.

No files require special attention; scripts/swebench_preflight.py and tests/test_swebench_preflight.py have small improvements suggested in comments.

Important Files Changed

Filename	Overview
scripts/swebench_preflight.py	New preflight validator — correctly guards isinstance/iid in all loops; main() has no try/except around load_patches, so malformed JSON gives a raw traceback rather than a clean error message.
tests/test_swebench_preflight.py	Good coverage for all gate paths; every call to _layout() leaks a temp directory since mkdtemp() is never cleaned up.
docs/GOLDEN_PATCH.md	New doc for golden-patch verification workflow; clear, accurate, no issues.
docs/HARNESS.md	New harness invariants doc with footgun table; content matches the implementation correctly.
README.md	Adds preflight step and links to new docs; straightforward addition, no issues.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    actor User
    participant main
    participant run_preflight
    participant load_patches
    participant validate_patches
    participant validate_run_scripts
    participant check_binary_patch_sections
    participant collect_known_warnings

    User->>main: --patches gold.json --scripts-dir run_scripts
    main->>main: check file/dir existence
    main->>run_preflight: patches_path, scripts_dir
    run_preflight->>load_patches: patches_path
    load_patches-->>run_preflight: list[dict] (or raises ValueError/JSONDecodeError)
    run_preflight->>validate_patches: patches
    validate_patches-->>run_preflight: patch_schema gate + errors
    run_preflight->>validate_run_scripts: patches, scripts_dir
    validate_run_scripts-->>run_preflight: run_scripts_layout gate + errors
    run_preflight->>check_binary_patch_sections: patches
    check_binary_patch_sections-->>run_preflight: binary_patch_sections gate (warn-only)
    run_preflight->>collect_known_warnings: patches
    collect_known_warnings-->>run_preflight: known warnings list
    run_preflight-->>main: "report {passed, gates, errors, warnings}"
    main-->>User: PASS/FAIL banner + gate details + ERR lines

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    actor User
    participant main
    participant run_preflight
    participant load_patches
    participant validate_patches
    participant validate_run_scripts
    participant check_binary_patch_sections
    participant collect_known_warnings

    User->>main: --patches gold.json --scripts-dir run_scripts
    main->>main: check file/dir existence
    main->>run_preflight: patches_path, scripts_dir
    run_preflight->>load_patches: patches_path
    load_patches-->>run_preflight: list[dict] (or raises ValueError/JSONDecodeError)
    run_preflight->>validate_patches: patches
    validate_patches-->>run_preflight: patch_schema gate + errors
    run_preflight->>validate_run_scripts: patches, scripts_dir
    validate_run_scripts-->>run_preflight: run_scripts_layout gate + errors
    run_preflight->>check_binary_patch_sections: patches
    check_binary_patch_sections-->>run_preflight: binary_patch_sections gate (warn-only)
    run_preflight->>collect_known_warnings: patches
    collect_known_warnings-->>run_preflight: known warnings list
    run_preflight-->>main: "report {passed, gates, errors, warnings}"
    main-->>User: PASS/FAIL banner + gate details + ERR lines

Loading

_{Reviews (3): Last reviewed commit: "fix(preflight): guard non-dict entries i..." | Re-trigger Greptile}