Repository navigation
fix(heartbeat): a node whose start-up registration failed registers when the storage returns (#646) - #651
Merged
Conversation
…when the storage returns, red (#646) Red (all three fail with TimeoutException against the unfixed service): StartAsync_RegistrationFailsOnce_NodeRegistersOnceStorageIsBack, StartAsync_RegistrationKeepsFailing_KeepsRetryingWithoutThrowing_AndStopsAtShutdown, Heartbeat_IsNotSentBeforeTheNodeIsRegistered_AndStartsAfterwards (its 'no heartbeat before registration' half is a guard).
…leave a ghost node, red (#646) Red: StopAsync_WhileARegistrationRetryIsInFlight_DeregistersAgainWhenItCompletes times out waiting for a second DeregisterServerAsync (found by the review of the retry).
…hen the storage returns (#646) The heartbeat timer was created only after a successful registration, so a storage error at start left the node out of the dashboard and the cluster tracking for the life of the process. The timer now always runs; until the node is registered each tick retries the registration and no heartbeat is sent. A retry that completes after the host stopped deregisters the node again.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
ServerHeartbeatServicecreated its heartbeat timer only after a successfulRegisterServerAsync. A storage error at start left the node out of the dashboard and the cluster tracking for the life of the process (the jobs still ran). The timer now always runs: until the node is registered each tick retries the registration (everyServerHeartbeatInterval, default 15 s) and no heartbeat is sent. A retry that completes after the host stopped deregisters the node again, so it cannot leave a ghost node. No storage contract change.Type of change
Red-first proof
14de1af: three tests fail withSystem.TimeoutExceptionagainst the unfixed service (the node must register once the storage answers,a node that is not registered keeps trying,heartbeats must start once the node is registered). The 'no heartbeat before registration' half of the third is a guard.StopAsyncderegistered. Its own red commit9492808:StopAsync_WhileARegistrationRetryIsInFlight_DeregistersAgainWhenItCompletestimes out waiting for a secondDeregisterServerAsync.Verification Checklist
dotnet format --verify-no-changespasseddotnet build -c Releasepassed, 0 warningsNexJob.Tests1215 passed on net8.0 and net10.0; core integration subset 157 passed on both;ServerHeartbeatServiceTestsrun repeatedly without flakesHeartbeatAsync_WhenStorageThrows_SurvivesAndLogsstill passes becauseHeartbeatAsyncis unchanged (the registered-or-not decision lives in the timer callback)src/NexJob/Internal/ServerHeartbeatService.cs) by the core owner--strict: 0 drift; delivery-guarantees and troubleshooting list the new warningCHANGELOG.mdupdated under[Unreleased]/ FixedNotes
RegisterServerAsyncis an upsert on every provider (SQL Server MERGE, PostgreSQL ON CONFLICT, MongoDB upsert, Redis hash set, InMemory overwrite), so overlapping retry ticks are harmless; the worst case is a duplicate Information log.StartedAtis now fixed at construction, so retries send the same value; the difference from before is milliseconds.Closes #646