Skip to content

fix: wait for the auxiliary supervisor when the bot shuts down [RAI-3134] - #1709

Open
JuaniRios wants to merge 1 commit into
rai-3133-hydration-incomplete-on-apply-failurefrom
rai-3134-wait-for-auxiliary-supervisor
Open

JuaniRios wants to merge 1 commit into
rai-3133-hydration-incomplete-on-apply-failurefrom
rai-3134-wait-for-auxiliary-supervisor

Conversation

@JuaniRios

@JuaniRios JuaniRios commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Part of RAI-3134.

On shutdown the bot now waits for the auxiliary supervisor to handle its stop, beside its other drains. Before, it sent the stop and returned at once. That supervisor runs the liq_ refreshes and the dashboard price feed, so they are now aborted before the telemetry flush and the clean-stop report run.

Live effect: next release. Shutdown waits for that supervisor for up to 90 s, then goes on with a warning · Risk: low (shutdown order; a stuck supervisor can delay shutdown by up to 90 s) · Ships: next bot release · Blocks: needs #1708 below it

Risks

  • The wait covers the supervisor ending, not each task unwinding. A P&L replay on a blocking thread can run on after it. Nothing waits for that.

Proof

  • A test drains each shutdown trigger with nothing else to wait for, and checks with one poll that the auxiliary supervisor already stopped. It fails when the wait is removed. Another test shows a supervisor that never stops cannot hold the drain past its timeout.
  • Not verified: a shutdown on a running bot.

@JuaniRios JuaniRios self-assigned this Oct 10, 2026
@linear-code

linear-code Bot commented Oct 10, 2026

Copy link
Copy Markdown

RAI-3134

This was referenced Oct 10, 2026
@JuaniRios
JuaniRios force-pushed the rai-3134-wait-for-auxiliary-supervisor branch from 99a311c to 23a6575 Compare October 11, 2026 05:31
@JuaniRios
JuaniRios force-pushed the rai-3133-hydration-incomplete-on-apply-failure branch from 0fcc771 to 1cd3b74 Compare October 11, 2026 05:31
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

No files to review.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

@JuaniRios
JuaniRios force-pushed the rai-3133-hydration-incomplete-on-apply-failure branch from 1cd3b74 to 6eb92c2 Compare October 11, 2026 07:03
@JuaniRios
JuaniRios force-pushed the rai-3134-wait-for-auxiliary-supervisor branch from 23a6575 to b681e69 Compare October 11, 2026 07:03
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

No files to review.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@rain-marvin

rain-marvin Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

🔎 Reviewing b681e69, started by @JuaniRios. The review will appear here when it's done.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes bot shutdown wait for the auxiliary supervisor (the metrics recorder upkeep, the liq_* refreshes and the dashboard price feed) instead of only sending it the stop. Every shutdown trigger, and a signal that arrives before startup completes, now goes through one drain_session helper. That helper sends the auxiliary stop and joins a wait on it, bounded by the same 90 s drain timeout, with the existing bot, server and detached-task drains. Because the waits run at the same time, the worst-case shutdown time does not grow.

I found no defects at this head. I checked the task-supervisor wait() and Shutdown handling: wait() reads a watch value, so a second call or a supervisor that already stopped returns at once, and an error only logs a warning. An error from the main drain still propagates after the auxiliary wait. The trigger test makes the auxiliary wait the only yield point (finished bot task, empty tracker, current-thread runtime) and checks it with one now_or_never poll, so it fails if the wait is removed. The stuck-supervisor test shows the bound holds. The change touches no stored state, config or shared signer, so upgrade and rollback are not affected. The fixes from all earlier threads are present, and the remaining limit (aborted tasks and a blocking P&L replay are not awaited) is stated in the doc comment and the PR's Risks section.

claude-opus-5-5 · high · 25 min

@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

@JuaniRios
JuaniRios marked this pull request as ready for review October 11, 2026 08:02
@rain-marvin

rain-marvin Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

🔎 Reviewing b681e69, started by @JuaniRios. The review will appear here when it's done.

@rain-marvin rain-marvin Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes the bot wait for the auxiliary supervisor (the liq_* refreshes and the dashboard price feed) during shutdown. Before, it only sent the stop. One new drain_session now handles every shutdown trigger and the signal-before-startup path. It sends the stop, then joins the existing bot, server and detached-task drains with a wait on the auxiliary supervisor. That wait uses the same drain timeout, so a stuck supervisor delays shutdown by that timeout at most and adds no second one. As a result, those tasks are aborted before the telemetry flush and the clean-stop report.

Overall: the change is small and correct, and the earlier threads are all fixed. The doc comments now describe only what the wait covers. every_shutdown_trigger_waits_for_the_auxiliary_supervisor makes the auxiliary wait the only yield point and checks it with a single now_or_never() poll, so it fails if the wait is removed. The paused-time test shows the timeout bounds a supervisor that never stops. One panel lane said the test runs on a multi-thread runtime and can pass without the wait. That is wrong: plain #[tokio::test] uses the current-thread runtime, so the finding was dropped. No upgrade or shared-stall risk: the change touches no persisted state, and the auxiliary wait runs beside the existing drains. The diff adds no sensitive data to this public repo.

claude-opus-5-5 · high · 52 min

@JuaniRios
JuaniRios force-pushed the rai-3134-wait-for-auxiliary-supervisor branch from b681e69 to afed2f1 Compare October 11, 2026 09:51
@JuaniRios
JuaniRios force-pushed the rai-3133-hydration-incomplete-on-apply-failure branch from 6eb92c2 to 0b4fe3a Compare October 11, 2026 09:51
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

@JuaniRios
JuaniRios force-pushed the rai-3134-wait-for-auxiliary-supervisor branch from afed2f1 to e4e303e Compare October 11, 2026 15:23
@JuaniRios
JuaniRios force-pushed the rai-3133-hydration-incomplete-on-apply-failure branch from 0b4fe3a to a4239d6 Compare October 11, 2026 15:23
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

@JuaniRios
JuaniRios force-pushed the rai-3133-hydration-incomplete-on-apply-failure branch from a4239d6 to f40853a Compare October 11, 2026 17:41
@JuaniRios
JuaniRios force-pushed the rai-3134-wait-for-auxiliary-supervisor branch from e4e303e to 56ebbf5 Compare October 11, 2026 17:41
@JuaniRios

Copy link
Copy Markdown
Collaborator Author

@coderabbitai approve

@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Comments resolved and changes approved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant