Skip to content

fix(heartbeat): a node whose start-up registration failed registers when the storage returns (#646) - #651

Merged
oluciano merged 3 commits into
developfrom
fix/646-node-registration-retry
Oct 19, 2026
Merged

oluciano merged 3 commits into
developfrom
fix/646-node-registration-retry

Conversation

@oluciano

@oluciano oluciano commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

ServerHeartbeatService created its heartbeat timer only after a successful RegisterServerAsync. A storage error at start left the node out of the dashboard and the cluster tracking for the life of the process (the jobs still ran). The timer now always runs: until the node is registered each tick retries the registration (every ServerHeartbeatInterval, default 15 s) and no heartbeat is sent. A retry that completes after the host stopped deregisters the node again, so it cannot leave a ghost node. No storage contract change.

Type of change

  • Bug fix

Red-first proof

  • Red commit 14de1af: three tests fail with System.TimeoutException against the unfixed service (the node must register once the storage answers, a node that is not registered keeps trying, heartbeats must start once the node is registered). The 'no heartbeat before registration' half of the third is a guard.
  • The review of the retry found a race: a retry in flight at shutdown could register after StopAsync deregistered. Its own red commit 9492808: StopAsync_WhileARegistrationRetryIsInFlight_DeregistersAgainWhenItCompletes times out waiting for a second DeregisterServerAsync.

Verification Checklist

  • dotnet format --verify-no-changes passed
  • dotnet build -c Release passed, 0 warnings
  • NexJob.Tests 1215 passed on net8.0 and net10.0; core integration subset 157 passed on both; ServerHeartbeatServiceTests run repeatedly without flakes
  • 3N: N1 registers when the storage returns, N2 keeps retrying without throwing and stops at shutdown, N3 no heartbeat before registration and no ghost after shutdown
  • No assertion removed or weakened; HeartbeatAsync_WhenStorageThrows_SurvivesAndLogs still passes because HeartbeatAsync is unchanged (the registered-or-not decision lives in the timer callback)
  • No storage interface or public API changed; protected core file changed (src/NexJob/Internal/ServerHeartbeatService.cs) by the core owner
  • DocsTruth --strict: 0 drift; delivery-guarantees and troubleshooting list the new warning
  • CHANGELOG.md updated under [Unreleased] / Fixed

Notes

  • RegisterServerAsync is an upsert on every provider (SQL Server MERGE, PostgreSQL ON CONFLICT, MongoDB upsert, Redis hash set, InMemory overwrite), so overlapping retry ticks are harmless; the worst case is a duplicate Information log.
  • StartedAt is now fixed at construction, so retries send the same value; the difference from before is milliseconds.
  • Patch scope: no new option, no contract change.

Closes #646

…when the storage returns, red (#646)

Red (all three fail with TimeoutException against the unfixed service): StartAsync_RegistrationFailsOnce_NodeRegistersOnceStorageIsBack, StartAsync_RegistrationKeepsFailing_KeepsRetryingWithoutThrowing_AndStopsAtShutdown, Heartbeat_IsNotSentBeforeTheNodeIsRegistered_AndStartsAfterwards (its 'no heartbeat before registration' half is a guard).
…leave a ghost node, red (#646)

Red: StopAsync_WhileARegistrationRetryIsInFlight_DeregistersAgainWhenItCompletes times out waiting for a second DeregisterServerAsync (found by the review of the retry).
…hen the storage returns (#646)

The heartbeat timer was created only after a successful registration, so a storage error at start left the node out of the dashboard and the cluster tracking for the life of the process. The timer now always runs; until the node is registered each tick retries the registration and no heartbeat is sent. A retry that completes after the host stopped deregisters the node again.
@sonarqubecloud

Copy link
Copy Markdown

@oluciano
oluciano merged commit 60ee221 into develop Oct 19, 2026
42 of 43 checks passed
@oluciano
oluciano deleted the fix/646-node-registration-retry branch October 19, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(heartbeat): a node whose start-up registration failed never registers when the storage returns

1 participant