fix(torghut): stabilize scheduler dependency health - #13011
Conversation
|
|
gregkonush
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b15ee54c8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| finally: | ||
| executor.shutdown(wait=False, cancel_futures=True) | ||
| with _protocol_health_lock: |
There was a problem hiding this comment.
When TigerBeetle is unresponsive and status requests overlap, the first caller can hold this global lock for the configured 5-second RPC timeout while every later caller waits without any deadline; after acquiring it, each waiter starts another full timeout. Three concurrent /trading/status reads can therefore exceed the 12-second _TradingStatusReadBudget, and a larger burst can occupy the shared FastAPI worker pool even after API proxies have timed out. Use a timed acquisition or share/cache the in-flight probe result so each request has an end-to-end bound, and cover the concurrent-outage case with a regression test.
AGENTS.md reference: AGENTS.md:L94-L96
Useful? React with 👍 / 👎.
4b15ee5 to
36c75a8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 36c75a86df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| finally: | ||
| executor.shutdown(wait=False, cancel_futures=True) | ||
| with _protocol_health_lock: | ||
| client = _protocol_health_client_for_settings(timeout_seconds) |
There was a problem hiding this comment.
On the first probe after startup, a failed probe, or a configuration change, this factory call runs synchronously before the timeout-protected client.nop(). RealTigerBeetleClient.__init__ performs socket.getaddrinfo and constructs ClientSync, so a slow Kubernetes DNS resolver or client initialization can now block /trading/status and dependency readiness beyond TORGHUT_TIGERBEETLE_HEALTH_TIMEOUT_SECONDS; the removed executor deadline previously bounded the whole operation. Construct the client within the same end-to-end deadline rather than applying the timeout only to the RPC.
Useful? React with 👍 / 👎.
36c75a8 to
1fea9cd
Compare
1 participant
Summary
/trading/statusrequest; apply the existing cancellable RPC deadline and discard failed clients before retry.Related Issues
PROOMPT-393
Testing
cd services/torghut && .venv/bin/pytest -q tests/test_tigerbeetle_client.py tests/test_tigerbeetle_status.py tests/test_process_roles.py— 60 passed.cd services/torghut && .venv/bin/pytest -q tests/test_tigerbeetle*.py tests/live_config_manifest_contract/test_single_writer_scheduler_manifest.py— 63 passed.cd services/torghut && .venv/bin/pyright -p pyrightconfig.json— 0 errors.cd services/torghut && .venv/bin/ruff check app/api/health_checks/tigerbeetle_health.py app/trading/tigerbeetle_client.py app/bootstrap.py app/scheduler_main.py tests/test_tigerbeetle_status.py tests/test_tigerbeetle_client.py— passed.cd services/torghut && .venv/bin/python scripts/run_pylint_torghut_quality_diff_gate.py --base e0036184dab3442218ff4e08c63d76ab89d1d6a9 app/api/health_checks/tigerbeetle_health.py app/bootstrap.py app/scheduler_main.py app/trading/tigerbeetle_client.py tests/test_tigerbeetle_client.py tests/test_tigerbeetle_status.py— passed.bun run lint:argocd— all rendered Argo resources passed kubeconform.sha256:e6448c4c476344eea647f4926872169c25adaa011a5d69c60dfc017cff8d05d9: TigerBeetle reachedclients=64/64with client eviction roughly every 5–6 seconds while scheduler threads remained stable at 74; natural cycle completions ranged from 35 to 169 seconds.Breaking Changes
None
Checklist
N/Awith justification).