[serve] Production defaults for HAProxy connect timeout and dead-replica detection

[harshit-anyscale]

Why this change

Two HAProxy defaults that are reasonable for a single-node dev cluster but poor once replicas come and go under autoscaling. Each is a default change only; both stay overridable through their existing environment variables.

Constant	Before	After
RAY_SERVE_HAPROXY_TIMEOUT_CONNECT_S	unset	5
RAY_SERVE_HAPROXY_OBSERVE_MARK_DOWN_ENABLED	0	1

timeout connect: unset → 5s

Replicas are in-cluster, so a connect that takes seconds means the node is gone rather than busy. Bounding it lets retry-on conn-failure + option redispatch reach another replica while the request still has budget. This is connection-establishment time only — it does not bound request duration.

timeout server is deliberately not set here; see Deliberately not included.

observe layer4 ... on-error mark-down: off → on

Live traffic marks a dead replica DOWN without waiting for a health checker, and redispatch plus the backup fallback take over. A false positive revives in ~0.5s via the existing health check. Backup/fallback servers are never observed.

Not a duplicate

Checked before opening, no overlapping work found:

gh pr list --repo ray-project/ray --state open --search "haproxy default"

The only open PR touching this area is #65698 (mine, direct-ingress listener close), which does not change any of these constants.

Tests

Two environments. (A) nightly ray-3.0.0.dev0 wheel with python/ray/serve and python/ray/tests symlinked in, pytest==7.4.4 per python/requirements/test-requirements.txt — used for the rendering and unit runs. (B) the same, plus a real HAProxy binary (brew 3.4.4) via RAY_SERVE_HAPROXY_BINARY_PATH, running the CI job's env (RAY_SERVE_ENABLE_HA_PROXY=1, RAY_SERVE_DIRECT_INGRESS_MIN_DRAINING_PERIOD_S=0.01, SERVE_SOCKET_REUSE_PORT_ENABLED=1) — used for the integration runs.

End-to-end with a real HAProxy (B) — the tests premerge flagged on earlier revisions of this PR, re-run at this revision's defaults with no env overrides:

test_autoscaling_policy.py::test_e2e_preserve_prev_replicas_rest_api   => passed (x2)
test_streaming_response.py                                             => 23 passed

Both were failing on earlier revisions and pass here. Caveats on this environment: it is HAProxy 3.4.4, while CI ships 2.8 via ray-haproxy; and macOS has no splice(2), so option splice-request / option splice-response had to be dropped from the rendered config for HAProxy to start locally. Those options affect data forwarding, not routing or timing. CI remains the authority.

HAProxy config rendering (A):

RAY_SERVE_ENABLE_HA_PROXY=1 pytest --pyargs ray.serve.tests.test_haproxy_api \
  -k "generate_config_file_internal or escapes_special_characters or close_spread_time
      or generate_backends_in_order or routers_and_targets or ingress_request_router_lua
      or router_servers_without_replica_ids or does_not_leak_into_other_backends
      or router_failure_503 or inherits_global_retry_policy or global_retry_knobs
      or default_data_plane_tuning"
=> 13 passed, 26 deselected

New test test_default_data_plane_tuning_renders pins the shipped values. Every other test in that file passes explicit overrides, so nothing previously caught a silent change to these defaults. It is skipif-guarded on the env vars it asserts about, so overriding any of them skips rather than fails:

RAY_SERVE_HAPROXY_OBSERVE_MARK_DOWN_ENABLED=0      => 1 skipped
RAY_SERVE_HAPROXY_TIMEOUT_SERVER_S=3600            => 1 skipped
RAY_SERVE_HAPROXY_TIMEOUT_CONNECT_S=9              => 1 skipped

The skip does not neuter it — changing a shipped default in source still fails the test.

Serve unit suite (A) — pytest python/ray/serve/tests/unit:

this branch:            195 failed, 4564 passed, 18 skipped, 2 errors
clean origin/master:    195 failed, 4564 passed, 18 skipped, 2 errors

Identical, so failure-neutral. The 195 are an artifact of running repo Serve code against a slightly older nightly core wheel on macOS; they are in test_deployment_state.py / test_handle_options.py and touch nothing this PR changes. Measured on the six-default revision of this PR; the diff has only shrunk since, so this is a ceiling rather than a current reading.

Lint at the versions pinned in .pre-commit-config.yaml, re-run at this revision:

ruff 0.8.4 check                => All checks passed
ruff 0.8.4 check --select I     => All checks passed
black 22.10.0 --check           => unchanged

mypy 1.7.0 and pyrefly 1.1.1 (both of which have constants.py on their allowlists) were clean on an earlier revision that already contained both remaining changes; they were not re-run at this exact commit, since everything removed since was a revert to master's own text.

One shape change worth calling out: RAY_SERVE_HAPROXY_TIMEOUT_CONNECT_S moves from the hand-rolled int(os.environ.get(...)) if os.environ.get(...) else ... conditional to the file's existing get_env_int_non_negative helper, which drops a # type: ignore[arg-type]. Its default is changing anyway, so the line is touched regardless, and the helper handles =0 correctly — the truthiness form would silently turn an explicit 0 into the default. It does mean an empty-string value now raises instead of being ignored, matching every other get_env_* constant in the file. Say the word if you would rather keep the original shape and just swap the else branch.

AI assistance

This change was drafted with AI assistance (Claude), including the code, the tests, and this description.

Opened as a draft on purpose: per AGENTS.md a human submitter has to review every changed line and run the relevant tests locally before review is requested. I have not finished that pass yet. Marking ready for review is the signal that it is done — please don't spend review time on it before then.

The binary-dependent HAProxy tests (26 deselected above) also still need a run on a machine that has haproxy installed.

[gemini-code-assist]

Code Review

This pull request updates several HAProxy default configurations in Ray Serve to optimize data plane tuning, including increasing the broadcast coalesce time, setting a default connection timeout of 5 seconds, enabling observe-mark-down by default, and switching the load balancing algorithm from leastconn to random(2). It also adds a new test to verify these default configurations. Feedback suggests simplifying the new test by removing a redundant mock.patch block, as the configuration file path is already explicitly passed to the HAProxyApi constructor.

[akyang-anyscale]

one comment on timeout connect. other change lgtm

[akyang-anyscale]

How does one disable this?

[harshit-anyscale]

Set it to 0. Documented in the comment as of fe769b1.

---

Details

get_env_int_non_negative accepts 0 (it validates >= 0), and the template gates on is not none, so RAY_SERVE_HAPROXY_TIMEOUT_CONNECT_S=0 renders timeout connect 0s. HAProxy stores an unset timeout as 0 internally, so that is indistinguishable from omitting the directive — in 2.8:

• proxy_parse_timeout accepts a literal 0 (only sub-millisecond non-zero values are rejected, as PARSE_TIME_UNDER: "minimum non-null value is 1 ms"), then assigns *tv = MS_TO_TICKS(0) == 0.
• The missing-timeouts warning tests !curproxy->timeout.connect (cfgparse.c), so 0 and unset take the same branch.
• The tarpit/queue inheritance is if (!timeout.queue) timeout.queue = timeout.connect, which copies the same 0 either way — so no divergent side effect from the fact that Serve leaves timeout queue unset.

Confirmed against a running HAProxy with a blackholed backend, config identical apart from this one line:

timeout connect 2s  ->  HTTP/1.1 503 Service Unavailable   after 2.00s
timeout connect 0s  ->  no response                        after 12.00s (connect still pending)

So =0 restores the previous behaviour exactly: infinite connect timeout, and the same missing timeouts for backend ... startup warning that omitting it produced.

Caveat on the runtime check: that was HAProxy 3.4.4 locally, since that is what I can run on macOS. The source references above are from 2.8, which is what ray-haproxy ships.

---

🤖 Investigated and drafted with Claude Code