Skip to content

feat(mcp): retry unhealthy servers and restore tools on recovery (#1655) - #1668

Merged
akramcodez merged 2 commits into
Nano-Collective:mainfrom
rishu685:feat/mcp-retry-unhealthy-servers
Oct 15, 2026
Merged

akramcodez merged 2 commits into
Nano-Collective:mainfrom
rishu685:feat/mcp-retry-unhealthy-servers

Conversation

@rishu685

@rishu685 rishu685 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #1655

  1. Continuous Health Checks on Unhealthy Servers (source/mcp/mcp-client.ts):

    • Updated startHealthChecks so servers marked unhealthy continue to be polled periodically on their configured healthCheckInterval.
    • Added isLivenessResponse to treat JSON-RPC error responses (e.g. -32601 Method not found) as proof that the server process is alive.
  2. Automatic Tool Restoration on Recovery (source/mcp/mcp-client.ts):

    • Added restoreServerHealth to re-discover tools, resources, and prompts from the server when a ping succeeds, invalidate tool mapping cache, and transition health back to connected.
    • Extended getToolEntries(forServerName?: string) to allow retrieving tool definitions for a specific server.
  3. Re-register Tools in ToolManager (source/tools/tool-manager.ts):

    • Handled { status: 'connected', serverName } health events in ToolManager to re-register restored tools in the active ToolRegistry.
  4. Documentation & Changeset:

    • Updated docs/configuration/mcp-configuration.md to reflect that unhealthy servers continue to be checked and restore their tools automatically upon response.
    • Added .changeset/mcp-retry-unhealthy-servers.md.
  5. Tests:

    • Added unit tests in source/mcp/mcp-client.spec.ts and source/tools/tool-manager.spec.ts covering unhealthy retry, tool restoration, -32601 liveness handling, and tool re-registration in ToolManager.

Verification

  • pnpm test:ava source/mcp/mcp-client.spec.ts source/tools/tool-manager.spec.ts (all 165 tests passed)
  • pnpm test:types (0 errors)
  • biome check (0 errors)
  • pnpm test:knip (0 issues)

@github-actions github-actions Bot added area:tools Tool implementations and tool-calling area:docs Documentation labels Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

nc-review: comments — 1 important

@rishu685 — a few things worth a look, none blocking.

PR #1668 implements MCP server recovery after unhealthy pings by re-polling at the configured interval, treating JSON-RPC -32601 MethodNotFound as proof of liveness, and re-registering tools on recovery. The change is well-scoped and the recovery logic and ToolManager re-registration are correct, but the issue's proposed solution explicitly calls for exponential backoff on unhealthy pings, which is not implemented.

🟠 important · completeness · source/mcp/mcp-client.ts:249

Issue #1655's proposed solution opens with 'Keep pinging unhealthy servers, with exponential backoff.' The diff removes the early-return guard so unhealthy servers are now polled, but it keeps the same healthCheckInterval (default 30s) for every retry. A server that is genuinely dead — the exact case the issue is trying to handle gracefully — is hit at the configured rate forever, with no widening between attempts. Either implement backoff (a few discrete steps that cap out at some multiple of the configured interval) or call out in the PR description that you intentionally traded simplicity for the issue's stated design, so a reviewer can sign off knowingly.


🔴 blocking · 🟠 a reviewer would ask for a change · ⚪ optional

Automated code review — correctness, security, design, tests, plus duplicates and scope. A human still decides; this is not a substitute for review and is not exhaustive. The required status checks separately cover lint, formatting, types, unused dependencies, the test suite and the build. This bot never merges. Maintainers can rerun with /re-review.

@github-actions github-actions Bot added the agent:comments nc-review left non-blocking findings label Oct 9, 2026
@rishu685

rishu685 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Updated the implementation to include exponential backoff for retrying unhealthy MCP servers:

  • Dynamic scheduling with exponential backoff (source/mcp/mcp-client.ts): Switched from a fixed setInterval to dynamic setTimeout self-scheduling. Consecutive health check failures scale the retry delay exponentially (baseInterval * 2^(attempts-1)), capped at 5 minutes (8x multiplier max).
  • Reset on recovery: Once an unhealthy server successfully responds to a ping (or returns a -32601 liveness response) and restores its tools, the failure counter resets and the polling interval returns to baseInterval.
  • Docs & Changeset: Updated docs/configuration/mcp-configuration.md and .changeset/mcp-retry-unhealthy-servers.md.
  • Tests: Added unit tests in source/mcp/mcp-client.spec.ts covering backoff delay escalation and recovery reset.

@akramcodez
akramcodez merged commit 4992bb7 into Nano-Collective:main Oct 15, 2026
17 checks passed
@akramcodez

Copy link
Copy Markdown
Member

Thanks for the PR @rishu685

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:comments nc-review left non-blocking findings area:docs Documentation area:tools Tool implementations and tool-calling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] MCP: retry an unhealthy server and restore its tools when it answers again

2 participants