Fix: raise docker client timeout so slow evals aren't silently scored unresolved

[robdmac]

docker.from_env() uses the SDK default 60s read timeout. On the --use_local_docker path, container.wait() then raises ReadTimeout for any test suite whose exec exceeds 60s (routine for Go/JS repos), so no output.json is written and the instance is silently scored unresolved.

Add DOCKER_CLIENT_TIMEOUT (default 3600s, overridable via env) and use it for the docker client. Modal path unaffected.

Fixes the false-negatives observed in #54, #22, #23.

Greptile Summary

This PR fixes silent scoring failures on --use_local_docker runs where test suites (especially Go/JS repos) take longer than the Docker SDK's default 60-second read timeout, causing container.wait() to raise ReadTimeout before output.json is written. The fix introduces a module-level DOCKER_CLIENT_TIMEOUT constant (default 3600 s, overridable via env) and passes it to docker.from_env().

• Adds DOCKER_CLIENT_TIMEOUT = int(os.environ.get("DOCKER_CLIENT_TIMEOUT", "3600")) as a named, overridable constant replacing the implicit SDK default.
• Passes the timeout to docker.from_env(timeout=DOCKER_CLIENT_TIMEOUT) so all Docker API calls on the local path — including container.wait() — use the extended timeout.

Confidence Score: 4/5

Safe to merge; the fix correctly targets the root cause of silent unresolved scoring on long-running test suites.

The change is minimal and well-scoped. The only subtlety is that the new timeout applies to every Docker API call on the client — including image pulls — not just container.wait(). A stalled registry pull would now block a worker for up to an hour instead of 60 s, but this is unlikely in practice and does not affect correctness.

swe_bench_pro_eval.py — specifically the images.pull() call that now shares the 3600 s timeout with container.wait()

Important Files Changed

Filename	Overview
swe_bench_pro_eval.py	Introduces DOCKER_CLIENT_TIMEOUT constant (default 3600s, env-overridable) and passes it to docker.from_env(); correctly extends the read timeout for container.wait() and all other Docker API calls on the local path. The broad timeout also applies to images.pull(), which is a minor tradeoff.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Eval as eval_with_docker()
    participant SDK as docker.from_env(timeout=3600)
    participant Daemon as Docker Daemon
    participant Container as Container

    Eval->>SDK: "docker.from_env(timeout=DOCKER_CLIENT_TIMEOUT)"
    SDK-->>Eval: client

    Eval->>Daemon: client.images.pull(image_uri)
    Daemon-->>Eval: image pulled

    Eval->>Daemon: "client.containers.run(..., detach=True, remove=True)"
    Daemon-->>Eval: container handle

    Eval->>Daemon: container.wait()
    note over Eval,Daemon: Blocks up to 3600s (was 60s)
    Daemon->>Container: runs entryscript.sh
    Container-->>Daemon: exit code
    Daemon-->>Eval: StatusCode

    Eval->>Eval: collect_outputs_local() → output.json written

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Eval as eval_with_docker()
    participant SDK as docker.from_env(timeout=3600)
    participant Daemon as Docker Daemon
    participant Container as Container

    Eval->>SDK: "docker.from_env(timeout=DOCKER_CLIENT_TIMEOUT)"
    SDK-->>Eval: client

    Eval->>Daemon: client.images.pull(image_uri)
    Daemon-->>Eval: image pulled

    Eval->>Daemon: "client.containers.run(..., detach=True, remove=True)"
    Daemon-->>Eval: container handle

    Eval->>Daemon: container.wait()
    note over Eval,Daemon: Blocks up to 3600s (was 60s)
    Daemon->>Container: runs entryscript.sh
    Container-->>Daemon: exit code
    Daemon-->>Eval: StatusCode

    Eval->>Eval: collect_outputs_local() → output.json written

Loading

Comments Outside Diff (1)

1. swe_bench_pro_eval.py, line 384-389 (link)

   The timeout passed to docker.from_env() is a socket-level read timeout that applies to all API calls on this client, including images.pull(). A hung or very slow registry pull will now block for up to 3600 s before raising an error, which may leave a worker silently stalled. Consider using a shorter, separate timeout for the pull call via docker.APIClient or by wrapping images.pull() with a threading.Timer, or at minimum document this tradeoff in the comment.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: swe_bench_pro_eval.py
   Line: 384-389
   
   Comment:
   The `timeout` passed to `docker.from_env()` is a socket-level read timeout that applies to **all** API calls on this client, including `images.pull()`. A hung or very slow registry pull will now block for up to 3600 s before raising an error, which may leave a worker silently stalled. Consider using a shorter, separate timeout for the pull call via `docker.APIClient` or by wrapping `images.pull()` with a `threading.Timer`, or at minimum document this tradeoff in the comment.
   
   
   
   How can I resolve this? If you propose a fix, please make it concise.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

   

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
swe_bench_pro_eval.py:384-389
The `timeout` passed to `docker.from_env()` is a socket-level read timeout that applies to **all** API calls on this client, including `images.pull()`. A hung or very slow registry pull will now block for up to 3600 s before raising an error, which may leave a worker silently stalled. Consider using a shorter, separate timeout for the pull call via `docker.APIClient` or by wrapping `images.pull()` with a `threading.Timer`, or at minimum document this tradeoff in the comment.

```suggestion
        # timeout applies to all SDK calls on this client (pull + wait).
        # Pull timeouts are less likely to be long, but if a registry stalls,
        # the worker will block for DOCKER_CLIENT_TIMEOUT seconds.
        client = docker.from_env(timeout=DOCKER_CLIENT_TIMEOUT)
        try:
            if docker_platform:
                client.images.pull(dockerhub_image_uri, platform=docker_platform)
            else:
                client.images.pull(dockerhub_image_uri)
```

_{Reviews (1): Last reviewed commit: "Fix: raise docker client timeout so slow..." | Re-trigger Greptile}