perf: investigate and reduce networking latency during onboard and validation

[ericksoa]

Problem

Multiple PRs have independently worked around slow networking by bumping timeouts, adding retries, and inserting sleep delays. Each fix addresses a symptom, but the cumulative effect is a fragile onboard experience that breaks on slower hardware and adds minutes of wall-clock time on every platform.

Examples:

• fix(onboard): widen inference verification timeouts on WSL2 (Fixes #987) #1998 / [Window x86 WSL2][Regression] NemoClaw/OpenShell inference verification reports false connectivity failure for NVIDIA Endpoints during onboarding #987: WSL2 inference validation timeouts doubled from 10s/15s → 20s/30s because DNS + TLS through the virtualized network stack is slow
• ARM64 gateway health polls: 5 polls × 2s → 30 polls × 10s (5 minutes) because k3s init takes longer
• WSL2 sandbox ready wait: 30 × 2s = 60s polling loop because pod init is slower under Docker Desktop
• Ollama model pull: 10-minute timeout
• Gateway start: exponential backoff with p-retry (10s min, factor 3)
• 30+ explicit sleep() calls scattered across onboard, recovery, and validation paths

As @brandonpelfrey noted in #1998: "Across multiple PRs from different folks I'm seeing things like add 10 seconds here and there so $thing doesn't time out. Want to make sure we're questioning why networking seems to be so fiddly." These timeout bumps are fragile — they work on the machines that were tested but will break on slower hardware.

Scope

This is a diagnostic and optimization effort, not a single bug fix. The goal is to understand why networking is slow and fix root causes rather than continuing to widen timeouts.

Phase 1: Diagnose

• Profile the onboard network path end-to-end on native Linux, macOS, and WSL2: where is time actually spent?
• Measure DNS resolution latency inside the k3s container vs. on the host — is CoreDNS adding overhead?
• Measure TLS handshake time for inference provider endpoints from inside the sandbox vs. from the host
• Determine whether the L7 proxy (gateway) adds measurable latency to inference validation probes
• Check if host.openshell.internal resolution is slow on WSL2 (goes through Windows DNS?)
• Profile the sleep() calls — which are covering real async settling vs. papering over race conditions?

Phase 2: Optimize

Based on diagnosis, potential fixes (non-exhaustive):

• DNS caching/preflight: Pre-resolve and cache provider DNS during onboard before validation probes hit the gateway
• Connection reuse: Validation probes currently spawn a new curl per attempt — a persistent connection (or at least keepalive) would skip repeated TCP+TLS handshakes
• Parallel health checks: Some sequential polling loops could overlap (e.g., gateway health + sandbox ready + dashboard ready)
• Reduce gateway round-trips: Validation probes go host → gateway → provider. If the gateway adds overhead, consider a direct probe option for validation only
• Replace sleeps with event-driven waits: Many sleep(2) calls are waiting for a process or pod state — replace with kubectl wait, readiness probes, or file watches where possible
• Platform-aware defaults: Instead of doubling timeouts for WSL2 as a special case, consider adaptive timeouts that measure the first probe latency and scale subsequent timeouts accordingly

Phase 3: Harden

• Add onboard timing telemetry (opt-in) so we can see real-world latency distributions
• Set a performance budget: onboard on a warm system with cached images should complete in < N seconds
• Add CI timing regression tests that fail if onboard wall-clock exceeds the budget

Current timeout inventory

Location	Current value	Why
Validation probe (onboard.ts)	10s connect / 15s total (20/30 on WSL2)	Inference provider reachability
HTTP probe default (http-probe.ts)	10s connect / 60s total	Streaming inference responses
Local provider health (local-inference.ts)	3s connect / 5s total	Ollama/vLLM liveness
Gateway liveness (nemoclaw.ts)	3s max	Quick gateway up/down check
ARM64 health poll	30 × 10s = 300s	k3s slow init on ARM
WSL2 sandbox ready	30 × 2s = 60s	Pod init under Docker Desktop
Ollama model pull	600s (10 min)	Large model download
OpenShell install	300s (5 min)	Binary download + extract

References

• fix(onboard): widen inference verification timeouts on WSL2 (Fixes #987) #1998 — WSL2 timeout widening (trigger for this investigation)
• [Window x86 WSL2][Regression] NemoClaw/OpenShell inference verification reports false connectivity failure for NVIDIA Endpoints during onboarding #987 — Original WSL2 inference verification timeout issue

[ericksoa]

First-Round Diagnostic Findings & First-Principles Recommendations

Full code audit of src/lib/onboard.ts, src/lib/http-probe.ts, src/lib/local-inference.ts, src/lib/platform.ts, src/lib/gateway-state.ts, src/lib/sandbox-create-stream.ts, src/lib/services.ts, src/lib/registry.ts, src/lib/nim.ts, and src/lib/agent-onboard.ts.

---

The core problem: "add more time" is an anti-pattern

The codebase has accumulated 14+ explicit sleep() calls (all in onboard.ts — the sleep() function itself is spawnSync("sleep", [seconds]), line 2097), 6 distinct polling loops with hardcoded iteration counts × fixed intervals, and platform-specific timeout multipliers (2× for WSL2, 6× for ARM64). Each was a reasonable local fix, but the cumulative result is a system that guesses how long things take instead of knowing when they're done.

The fundamental issue isn't that timeouts are too short or too long — it's that most of these waits shouldn't be time-based at all.

---

Inventory: What's actually waiting, and why

Category A: Sleeps papering over race conditions (~7 instances)

These sleep() calls exist because something was spawned and the code needs it to be ready, but instead of checking readiness, it just waits a fixed duration:

Location	Sleep	What it's really waiting for
onboard.ts:1791	sleep(1)	Ollama auth proxy to bind its port after killing stale process
onboard.ts:1807	sleep(1)	Newly spawned auth proxy to accept connections
onboard.ts:1832	sleep(1)	Auth proxy restart after host reboot recovery
onboard.ts:3988	sleep(2)	ollama serve to bind port (Linux/WSL)
onboard.ts:4056	sleep(2)	ollama serve to bind port (macOS after brew install)
onboard.ts:2673	sleep(5)	CoreDNS to process a patch
onboard.ts:4680-5340	sleep(2) ×3 retry loops	Policy preset apply — "sandbox not found" eventual consistency

First-principles fix: Replace every one of these with a readiness probe on the actual resource:

• Proxy port binding: waitForPort(port, {timeout: 10_000}) — a tight loop attempting TCP connect to localhost:port. Resolves in milliseconds once the port is bound, vs. always paying 1-2s.
• Ollama serve: Same — probe http://127.0.0.1:11434/api/tags (the existing health endpoint in local-inference.ts:137) until it returns, with a deadline.
• CoreDNS patch: Use kubectl wait --for=condition=ready pod -l k8s-app=kube-dns --timeout=15s instead of a blind 5s sleep. CoreDNS could be ready in 500ms or need 10s; a fixed sleep handles neither well.
• Policy preset visibility: The "sandbox not found" retries are masking Kubernetes propagation delay. Use kubectl wait --for=jsonpath='{.status.phase}'=Running pod/<name> --timeout=30s before applying presets. Alternatively, call openshell sandbox list once with a short backoff, but only retry on the specific "not found" error, not unconditionally.

Category B: Polling loops with fixed intervals (~6 loops)

These are better than blind sleeps (they at least check a condition), but they use fixed intervals that waste time on fast systems and can still timeout on slow ones:

Location	Parameters	Max wait	What it polls
Gateway health (onboard.ts:2600-2612)	5×2s / 30×10s (ARM64)	10s / 300s	openshell status + gateway info
Sandbox ready (onboard.ts:3352-3358)	30×2s	60s	openshell sandbox list
Dashboard ready (onboard.ts:3381-3393)	15×2s	30s	curl -sf http://localhost:3000/
Gateway recovery (onboard.ts:2726-2740)	10×2s	20s	openshell status
Agent gateway (agent-onboard.ts:168-184)	~20×3s	60s	Health probe URL
Sandbox create stream (sandbox-create-stream.ts:186-213)	pollIntervalMs=2000	unbounded	readyCheck callback

First-principles fix: Deadline-based polling with adaptive backoff.

Instead of for (let i = 0; i < N; i++) { check(); sleep(interval); }, use:

async function waitUntil(check: () => boolean, opts: {
  deadline: number,       // absolute wall-clock deadline
  initialInterval: number, // start fast (e.g. 200ms)
  maxInterval: number,     // cap backoff (e.g. 5s)
  factor: number,          // e.g. 1.5
}): Promise<boolean> {
  let interval = opts.initialInterval;
  while (Date.now() < opts.deadline) {
    if (check()) return true;
    await sleep(Math.min(interval, opts.deadline - Date.now()));
    interval = Math.min(interval * opts.factor, opts.maxInterval);
  }
  return false;
}

Benefits:

• Fast systems finish fast: If the gateway is healthy in 400ms, you detect it at ~400ms instead of waiting 2s.
• Slow systems get the full budget: The deadline is the constraint, not an arbitrary iteration count.
• No platform multiplication needed: A 120s deadline works for both x86 (finishes in 2s) and ARM64 (finishes in 90-180s). The same code handles both without isArm64 ? 30 : 5 branching.
• Adaptive first-probe calibration: Optionally, measure the first successful check's latency and use it to set expectations for subsequent checks. If gateway health takes 90s, sandbox readiness probably won't be instant either.

Category C: Validation probes with no connection reuse

Every inference validation probe spawns a fresh curl subprocess via spawnSync() (http-probe.ts:100). Each invocation:

1. Forks a process
2. Does a fresh DNS lookup
3. Opens a new TCP connection
4. Performs a full TLS handshake
5. Sends the HTTP request
6. Tears everything down

The probe sequence in probeOpenAiLikeEndpoint() (onboard.ts:1322) tries up to 4 sequential probes: Responses API → Streaming event check → Chat Completions → Retry with doubled timeouts. That's 4× the DNS + TCP + TLS overhead.

On WSL2, where TLS handshakes are already slow enough to require 20s connect timeouts, this compounds badly.

First-principles fix options (in order of impact/difficulty):

1. DNS pre-resolution: Before the first probe, resolve the endpoint hostname once and pass the IP to curl via --resolve host:port:ip. This eliminates repeated DNS lookups and is a one-line change per probe call. Biggest bang for the buck on WSL2 where DNS goes through the Windows resolver.

2. Connection reuse via --next: Curl supports --next to send multiple requests on the same connection. Chain the Responses API and Chat Completions probes into a single curl invocation. This amortizes the TCP+TLS cost across all probes.

3. Replace curl subprocesses with native HTTP: Node.js's built-in http/https modules support keepalive agents. A single https.Agent({keepAlive: true}) would let all probes in a sequence share one connection. This is a bigger refactor but eliminates process spawning overhead entirely.

4. Parallel probes where possible: The Responses API probe and the Chat Completions probe are currently sequential with fallback logic. But the initial connectivity check could be a lightweight HEAD or OPTIONS request to establish the connection, then run the real probes.

Category D: Sequential orchestration where parallelism is safe

The onboard flow is almost entirely sequential:

Gateway start → Gateway health poll → Provider selection (interactive) →
Inference validation probes → Sandbox creation → Sandbox ready poll →
Dashboard ready poll → Policy application → DNS proxy setup

Some of these have genuine dependencies. But some don't:

Opportunity	Currently	Could be
Sandbox ready + Dashboard ready	Sequential (60s + 30s = 90s max)	Parallel (max 60s) — dashboard lives inside the sandbox, so if the sandbox is ready, start probing dashboard immediately in parallel with registration
DNS pre-resolution	Not done	During provider selection — while the user is answering interactive prompts, resolve the inference endpoint DNS in the background
Ollama model pull + Gateway start	Sequential	Parallel on local-inference paths — model pull doesn't depend on gateway

The interactive prompts (provider selection, model selection) are dead time from the system's perspective. Use it.

---

Platform-specific timeout widening: the symptom, not the disease

The WSL2 timeout doubling (onboard.ts:1259-1264) and ARM64 poll expansion (onboard.ts:2596-2599) are classic symptoms of the "add more time" anti-pattern. They work on the machines that were tested but are fragile:

• WSL2 with a corporate VPN might need 3× not 2×
• A fast ARM64 Mac needs 2s not 300s for gateway health
• A slow x86 VM on a congested network might need WSL2-class timeouts despite not being WSL2

First-principles alternative: adaptive timeouts.

1. Calibration probe: At the start of onboard, make one trivial HTTP request (e.g., HEAD to the inference endpoint, or curl -sf https://api.example.com/health) and measure the round-trip time.
2. Scale all subsequent timeouts as a multiple of the calibration RTT. If the calibration takes 50ms, set connect timeout to 5s (100×). If it takes 3s, set connect timeout to 30s. This automatically adapts to WSL2, VPNs, ARM64, slow VMs, and any future platform without hardcoded branches.
3. Fall back to generous defaults if calibration itself fails (network truly unreachable).

This eliminates every isWsl() and isArm64 timeout branch and replaces them with a single, principled mechanism.

---

The sleep() implementation itself is wasteful

function sleep(seconds) {
  require("child_process").spawnSync("sleep", [String(seconds)]);
}

This spawns an entire OS process just to wait. On macOS/Linux this forks a process, loads the sleep binary, and blocks the Node.js event loop. For sub-second waits, Atomics.wait() (already used in registry.ts:104) is better. For async code, setTimeout with a promise wrapper is better. The registry code already demonstrates the right pattern.

---

Recommended implementation plan

Phase 1 — Quick wins (low risk, high impact):

1. waitForPort() utility — Replace the 5 proxy/Ollama sleep() calls with TCP connect probes. ~50 lines of new code, eliminates ~8s of unconditional wait.
2. DNS pre-resolution for validation probes — Add --resolve to curl args after a single dns.resolve() call. ~20 lines, eliminates repeated DNS on WSL2.
3. Parallelize sandbox ready + dashboard ready polls — Use Promise.all() or run both checks in the same polling loop. ~15 lines changed, saves up to 30s.

Phase 2 — Structural improvements (medium risk, high impact):

4. waitUntil() deadline-based polling — Replace all 6 polling loops with a single utility that uses adaptive backoff. Eliminates platform-specific iteration count/interval branching.
5. Adaptive timeout calibration — One calibration probe at onboard start, scale all subsequent timeouts. Eliminates isWsl() and isArm64 timeout branches.
6. Connection reuse for validation probes — Either --resolve + --next in curl, or migrate to native Node.js HTTP client with keepalive.

Phase 3 — Instrumentation (enables future optimization):

7. Onboard timing telemetry — Wrap each phase in performance.mark()/performance.measure() and emit a summary at the end. Shows users (and us) where time is actually spent.
8. Performance budget in CI — Set a wall-clock target for onboard on a warm system, fail CI if exceeded.

---

Key code references

File	Lines	What
src/lib/onboard.ts	2097-2099	sleep() implementation (spawnSync)
src/lib/onboard.ts	1259-1264	getValidationProbeCurlArgs() — WSL2 timeout doubling
src/lib/onboard.ts	2596-2612	Gateway health polling loop (ARM64 branching)
src/lib/onboard.ts	3346-3395	Sandbox + dashboard readiness polling
src/lib/onboard.ts	1322-1509	probeOpenAiLikeEndpoint() — sequential probe chain
src/lib/onboard.ts	1453-1480	Retry with doubled timeouts on failure
src/lib/onboard.ts	1791, 1807, 1832	Ollama proxy sleep(1) calls
src/lib/onboard.ts	3988, 4056	Ollama serve sleep(2) calls
src/lib/http-probe.ts	94-155	runCurlProbe() — subprocess-per-probe
src/lib/local-inference.ts	126-154	Local provider health check
src/lib/registry.ts	104	Atomics.wait() — better sleep pattern
src/lib/platform.ts	8-29	isWsl() detection

---

Summary

The onboard/validation latency isn't primarily a networking problem — it's a waiting strategy problem. The network is as fast as it is; the code just doesn't know when to stop waiting. Replacing guessed durations with observed readiness, reusing connections instead of re-establishing them, and running independent checks in parallel would cut worst-case onboard time significantly without adding a single timeout.

[ericksoa]

Design: Systemic Latency Elimination

Based on the diagnostic findings above, here's the proposed design to fix this systemically rather than continuing to widen timeouts.

---

New Module 1: src/lib/wait.ts — Unified Waiting Framework

The single replacement for every sleep(), polling loop, and retry pattern.

Core API:

interface WaitOptions {
  deadlineMs: number;          // absolute wall-clock deadline (Date.now() + budget)
  initialIntervalMs?: number;  // start fast, back off (default: 150ms)
  maxIntervalMs?: number;      // cap backoff (default: 5000ms)
  factor?: number;             // backoff multiplier (default: 1.5)
  label?: string;              // for logging/telemetry
  signal?: AbortSignal;        // cancellation
}

function waitUntil(check: () => boolean, opts: WaitOptions): boolean;
function waitForPort(port: number, host: string, opts: WaitOptions): boolean;
function waitForHttp(url: string, opts: WaitOptions): boolean;
function sleepMs(ms: number): void;   // Atomics.wait, no subprocess
function sleepSec(seconds: number): void;

Key principle: deadline-based, not iteration-based. waitUntil(check, {deadlineMs: Date.now() + 120_000}) gives slow systems the full budget while fast systems exit immediately. No more isArm64 ? 30 : 5 branching.

Adaptive backoff starting fast. Initial interval of 150ms means a port that binds in 50ms is detected within 200ms, not after a 2-second sleep. Backs off to 5s max.

Atomics.wait() for sleep. Replaces spawnSync("sleep", [seconds]) — no process fork. Already proven in registry.ts:104.

What it replaces:

Current code	Replacement
sleep(1) after proxy spawn (×3)	waitForPort(PROXY_PORT, "127.0.0.1", {deadlineMs: now+10_000})
sleep(2) after ollama serve (×2)	waitForHttp("http://127.0.0.1:11434/api/tags", {deadlineMs: now+15_000})
sleep(5) after CoreDNS patch	waitUntil(corednsReady, {deadlineMs: now+15_000})
Gateway health: 5×2s / 30×10s	waitUntil(isGatewayHealthy, {deadlineMs: now+120_000})
Sandbox ready: 30×2s	waitUntil(isSandboxReady, {deadlineMs: now+120_000})
Dashboard ready: 15×2s	waitForHttp(dashboardUrl, {deadlineMs: now+60_000})
Gateway recovery: 10×2s	waitUntil(isConnected, {deadlineMs: now+30_000})
Policy preset retries: 3×sleep(2)	waitUntil(presetApplySucceeds, {deadlineMs: now+30_000})

---

New Module 2: src/lib/net-calibrate.ts — Adaptive Timeout Calibration

Replaces all isWsl() and isArm64 timeout branches with measured network conditions.

interface NetworkProfile {
  dnsMs: number;              // measured DNS resolution time
  connectMs: number;          // measured TCP connect time
  tlsMs: number;              // measured TLS handshake time
  firstByteMs: number;        // total first-byte latency
  platform: string;           // for logging only, not branching
  connectTimeoutMs: number;   // recommended: max(5000, connectMs * 10)
  requestTimeoutMs: number;   // recommended: max(15000, firstByteMs * 5)
  pollDeadlineMs: number;     // recommended: max(30000, firstByteMs * 20)
}

function calibrate(endpointUrl: string): NetworkProfile;
function defaultProfile(): NetworkProfile;  // fallback if calibration fails

How it works: Single curl --write-out '%{time_namelookup} %{time_connect} %{time_appconnect}' call. Extracts DNS, TCP, TLS timings. Scales all subsequent timeouts as multiples of observed latency.

What it eliminates:

• getValidationProbeCurlArgs() with its isWsl() branch (onboard.ts:1259-1264)
• isArm64 poll count/interval branches (onboard.ts:2596-2599)
• Retry-with-doubled-timeouts hack (onboard.ts:1453-1480)
• NEMOCLAW_HEALTH_POLL_COUNT / NEMOCLAW_HEALTH_POLL_INTERVAL env vars
• The WSL2 hint message (onboard.ts:1499-1503)

Calibration timing: Runs once after gateway health confirmed, before validation probes. ~1-3s.

---

New Module 3: src/lib/probe-pool.ts — Connection-Reused Probes

interface ProbePool {
  preResolve(hostname: string): string | null;
  probe(argv: string[], url: string, opts?: CurlProbeOptions): CurlProbeResult;
  streamingProbe(argv: string[], url: string, opts?: CurlProbeOptions): StreamingProbeResult;
  getTimingArgs(): string[];
}

function createProbePool(opts?: { resolveMap?: Map<string,string>; profile?: NetworkProfile }): ProbePool;

How it works:

• On first probe for a hostname, resolves DNS and caches the IP
• Injects --resolve hostname:port:ip into all subsequent curl calls
• Uses NetworkProfile for timeout values instead of hardcoded constants
• Backward compatible: runCurlProbe() gets optional pool parameter

Impact: 4 sequential probes to same endpoint → DNS resolved once, not 4×. Critical on WSL2 where DNS goes through the Windows resolver.

---

New Module 4: src/lib/onboard-timing.ts — Instrumentation

function createTimer(): {
  start(label: string): void;
  end(label: string): void;
  report(): OnboardTimingReport;
};

Wraps each onboard step. Prints summary on completion:

Onboard completed in 47s
  preflight       3.2s
  gateway        12.4s
  provider        8.1s  (user prompt)
  inference       2.3s
  sandbox        11.8s
  openclaw        1.1s
  policies        2.1s
Network: dns=45ms tcp=12ms tls=89ms

Controlled by NEMOCLAW_TIMING=1 (off by default, always on in CI).

---

Orchestration Changes in src/lib/onboard.ts

Parallel step execution:

CURRENT:  [Preflight] → [Gateway] → [Provider] → [Inference] → [Messaging] → [Sandbox] → ...

PROPOSED: [Preflight] → [Gateway start ←―――――――――――――――――――――――→]
                        [Provider + Messaging prompts (parallel)]  → [Inference] → [Sandbox] → ...
                              ↑ DNS pre-resolve in background ↑

• Gateway start runs as async background operation while user answers prompts
• DNS pre-resolution happens during user prompt dead time (free parallelism)
• Sandbox-ready and dashboard-ready polls run in parallel (saves up to 30s)

---

Phased PR Plan

Each PR is independently shippable and testable:

PR 1: wait.ts foundation + replace sleep() primitive

• New: src/lib/wait.ts, test/wait.test.js
• Swap spawnSync("sleep") → sleepMs() import (behavioral no-op, better implementation)
• Risk: Low — same behavior, no subprocess spawned

PR 2: Replace race-condition sleeps with readiness probes

• sleep(1) after proxy → waitForPort()
• sleep(2) after Ollama → waitForHttp(ollamaHealthUrl)
• sleep(5) after CoreDNS → waitUntil(corednsReady)
• Policy preset retries → waitUntil(presetApplySucceeds)
• Risk: Low-medium — each is a strict improvement; test on WSL2 + ARM64

PR 3: Replace polling loops with deadline-based waits

• All 6 polling loops → waitUntil() with deadlines
• Delete isArm64 poll count/interval branches
• Delete NEMOCLAW_HEALTH_POLL_COUNT / NEMOCLAW_HEALTH_POLL_INTERVAL env vars
• Risk: Medium — changes observable timing; needs E2E validation

PR 4: Network calibration + adaptive timeouts

• New: src/lib/net-calibrate.ts, test/net-calibrate.test.js
• Delete getValidationProbeCurlArgs() and its isWsl() branch
• Delete retry-with-doubled-timeouts hack
• Risk: Medium — changes timeout values for all platforms; needs WSL2 + ARM64 testing

PR 5: Probe connection pooling

• New: src/lib/probe-pool.ts, test/probe-pool.test.js
• Add optional pool parameter to runCurlProbe() / runStreamingEventProbe()
• DNS pre-resolution + --resolve injection
• Risk: Low — additive, backward compatible

PR 6: Parallel orchestration

• Gateway start concurrent with user prompts
• Sandbox-ready ∥ dashboard-ready
• DNS pre-resolution during prompt dead time
• Risk: Medium-high — changes control flow; needs thorough E2E

PR 7: Timing instrumentation

• New: src/lib/onboard-timing.ts
• Wrap each step, print summary behind NEMOCLAW_TIMING=1
• CI job with wall-clock performance budget
• Risk: Low — observability only

---

What Gets Deleted

Code	Location	Replaced by
spawnSync("sleep", [seconds]) implementation	onboard.ts:2097, agent-onboard.ts:102	sleepMs() via Atomics.wait
sleep(1) after proxy (×3)	onboard.ts:1791, 1807, 1832	waitForPort()
sleep(2) after ollama serve (×2)	onboard.ts:3988, 4056	waitForHttp()
sleep(5) after CoreDNS	onboard.ts:2673	waitUntil(corednsReady)
sleep(2) in preset retries (×3 loops)	onboard.ts:4680-5340	waitUntil(presetApplySucceeds)
getValidationProbeCurlArgs() + isWsl branch	onboard.ts:1259-1264	pool.getTimingArgs() from calibration
Retry-with-doubled-timeouts hack	onboard.ts:1453-1480	Calibrated timeouts
isArm64 poll count/interval branches	onboard.ts:2596-2599	Deadline-based waitUntil()
NEMOCLAW_HEALTH_POLL_COUNT env var	onboard.ts:2598	Deadline parameter
NEMOCLAW_HEALTH_POLL_INTERVAL env var	onboard.ts:2599	Adaptive backoff
retriedAfterTimeout + WSL hint	onboard.ts:1457, 1499-1503	Calibrated timeouts
Gateway health for-loop	onboard.ts:2600-2612	waitUntil(isGatewayHealthy, {deadlineMs})
Sandbox ready for-loop	onboard.ts:3352-3358	waitUntil(isSandboxReady, {deadlineMs})
Dashboard ready for-loop	onboard.ts:3381-3393	waitForHttp(dashboardUrl, {deadlineMs})
Recovery poll for-loop	onboard.ts:2726-2740	waitUntil(isConnected, {deadlineMs})

Total: ~150 lines of dead code eliminated, replaced by ~200 lines of framework + ~50 lines of integration.

[ericksoa]

Regression Risk Analysis

Deep analysis of every external surface the design plan touches. Organized by risk level.

---

RISK: HIGH — Tests that will break and must be rewritten

1. test/wsl2-probe-timeout.test.ts — entire file becomes invalid

This test file directly imports and asserts against getValidationProbeCurlArgs, which the plan deletes.

What breaks:

• Lines 9, 12-38: Import and test getValidationProbeCurlArgs() by name — the function ceases to exist
• Lines 14-28: Assert exact timeout values ["--connect-timeout", "10", "--max-time", "15"] and the WSL2 variants — these hardcoded values are replaced by calibrated values
• Lines 41-80: Assert retry-with-doubled-timeouts by scanning compiled source for isTimeoutOrConnFailure, String(Number(arg) * 2), WSL2 detected, --skip-verify — all of this code is deleted
• Line 9: import { getValidationProbeCurlArgs } from "../dist/lib/onboard" — export removed from onboard.ts:6151

Mitigation: Replace this test file entirely. New tests should:

• Test net-calibrate.ts — verify calibrate() computes correct timeout multipliers from mock curl --write-out output
• Test probe-pool.ts — verify getTimingArgs() returns calibrated values
• Test that calibration failure falls back to defaultProfile() generous defaults
• Drop the source-scanning tests (lines 41-80) — these are brittle by nature

PR timing: Must ship with PR 4 (adaptive calibration) — can't delete the function without providing the replacement test.

---

2. test/onboard.test.ts:1028-1079 — gateway failure test depends on NEMOCLAW_HEALTH_POLL_COUNT=0

What breaks:

• Line 1063: NEMOCLAW_HEALTH_POLL_COUNT: "0" — env var is deleted in PR 3
• Lines 1028-1031: Test comment explicitly documents the dependency: "NEMOCLAW_HEALTH_POLL_COUNT=0 skips the health-poll loop so the function throws 'Gateway failed to start' on the first attempt"

What this test actually needs: A way to make the gateway health check fail immediately, without waiting. The env var was a test-speed hack, not a user feature.

Mitigation: The new waitUntil() deadline-based approach needs a test-friendly escape hatch:

• Option A: NEMOCLAW_HEALTH_DEADLINE_MS=0 env var — deadline in the past, check runs once, fails immediately. Preserves the test pattern.
• Option B: Inject the waitUntil implementation via dependency injection (pass a mock that always returns false).
• Recommended: Option A — minimal test change, same pattern (env var = 0 means "don't wait").

---

3. test/cli.test.ts:25-26 — every CLI test uses both env vars to avoid polling delays

What breaks:

NEMOCLAW_HEALTH_POLL_COUNT: "1",
NEMOCLAW_HEALTH_POLL_INTERVAL: "0",

These are injected into every runWithEnv() call. Without them, CLI tests that trigger gateway-touching code paths will hang for 10-300 seconds.

Mitigation: Replace with the new equivalent:

• NEMOCLAW_HEALTH_DEADLINE_MS: "100" (or "0") — same intent, new mechanism
• Or if waitUntil() uses a separate env var for test override, use that

PR timing: Must ship with PR 3 (deadline-based waits) — tests break the moment the old env vars are removed.

---

RISK: MEDIUM — E2E tests that parse console output

4. E2E tests parse [resume] Skipping messages

Files affected:

• test/e2e/test-onboard-resume.sh:229-241 — greps for [resume] Skipping preflight (cached), [resume] Skipping gateway (running), [resume] Skipping sandbox (${SANDBOX_NAME})
• test/e2e/test-onboard-repair.sh:198-204 — same pattern

Risk assessment: The plan does NOT change step names or resume skip messages. The parallel orchestration (PR 6) changes the order steps execute but not the messages emitted. Low risk as long as step names are preserved.

Mitigation: None needed for PRs 1-5. For PR 6 (parallel orchestration), verify that resume messages still appear in the same order in stdout. If gateway and provider run concurrently, their "Skipping" messages may interleave. Test this explicitly.

---

5. E2E tests check sandbox readiness

• test/e2e/test-sandbox-survival.sh:493 — greps openshell sandbox list for running|ready
• test/e2e/test-messaging-providers.sh:240 — "Verify sandbox is ready"

Risk: These don't depend on NemoClaw's polling implementation — they run openshell sandbox list directly. No risk.

---

RISK: MEDIUM — Behavioral contracts with external consumers

6. NEMOCLAW_HEALTH_POLL_COUNT / NEMOCLAW_HEALTH_POLL_INTERVAL as a user-facing tuning surface

Current state: These env vars are NOT documented in docs/. They appear only in source code and tests. However:

• Users may have discovered them through source reading or community posts
• ARM64 users (Raspberry Pi) may have set custom values
• CI/automation scripts may use them to control timing

Mitigation options:

• Option A (aggressive): Delete them. Add NEMOCLAW_HEALTH_DEADLINE_MS as the replacement. Document it.
• Option B (conservative): Keep them as deprecated aliases. If NEMOCLAW_HEALTH_POLL_COUNT is set, convert: deadline = count * interval * 1000. Log a deprecation warning. Remove in next major version.
• Recommended: Option B for PR 3, with removal tracked for a future PR. This is the one place where "just delete it" carries real regression risk for external consumers we can't see.

---

7. The WSL2 hint message in error output

WSL2 detected — network verification may be slower than expected.
Run `nemoclaw onboard` with the `--skip-verify` flag if this endpoint is known to be reachable.

Analysis: The --skip-verify flag doesn't actually exist as a CLI argument. The hint references a feature that was never implemented. It directs users to a dead end. There are provider configs with skipVerify: true internally, but no CLI surface.

Mitigation: Deleting this message is a net improvement — it stops directing users to a nonexistent flag. The adaptive calibration should make the message unnecessary anyway. If a calibrated timeout still fails, the error message should say something actionable (like "check your network connection" or "try again").

---

8. Sandbox creation timeout message: "did not become ready within 60s"

The hardcoded "60s" in the error message (onboard.ts:3366) becomes incorrect if the deadline is no longer 60s.

Mitigation: Change the message to include the actual deadline: "did not become ready within ${deadlineSec}s". Trivial fix, ship with PR 3.

---

RISK: LOW — Internal changes with no external surface

9. sleep() implementation swap (PR 1)

Replacing spawnSync("sleep", [seconds]) with Atomics.wait() is purely internal. No test asserts on the sleep mechanism. No external consumer can tell the difference.

Risk: Near-zero. Only concern: Atomics.wait() requires a SharedArrayBuffer, which needs the buffer to exist. The pattern in registry.ts:104 already proves this works in the codebase.

---

10. Probe pool / DNS pre-resolution (PR 5)

Adding optional pool parameter to runCurlProbe() is backward-compatible — existing callers pass no pool and behave identically.

Risk: Zero for existing callers. New behavior only activates when a pool is explicitly passed.

---

11. Timing instrumentation (PR 7)

Output only appears when NEMOCLAW_TIMING=1. Default is off. No existing behavior changes.

Risk: Zero.

---

12. Session resume compatibility

The session file (~/.nemoclaw/onboard-session.json) stores step names and completion status but NO timeout values, poll counts, or intervals. Changing the waiting mechanism does not change what gets persisted.

Risk: Zero for PRs 1-5. PR 6 (parallel orchestration) needs care: if a session was saved mid-onboard under the old sequential flow, resuming under the new parallel flow must still work. The current resume logic checks step completion, not order, so this should be safe — but must be tested explicitly.

---

13. agent-onboard.ts has its own independent sleep/polling

agent-onboard.ts:102-104 defines its own sleep() function and agent-onboard.ts:168-184 has its own polling loop. These do NOT use NEMOCLAW_HEALTH_POLL_* env vars — they use probe.timeout_seconds from agent definitions.

Mitigation: PR 1 replaces the sleep primitive. PR 3 replaces the polling loop with waitUntil(). These are independent of the main onboard changes. The agent gateway health timeout is derived from the agent definition, not env vars, so no external surface changes.

---

14. k8s/nemoclaw-k8s.yaml has its own sleep calls

Lines 62, 68: sleep 1 and sleep 2 in the init container script waiting for Docker. These are in shell, not TypeScript, and are NOT affected by any of the proposed changes.

Risk: Zero. These are a separate concern (Kubernetes init container, not onboard CLI).

---

Summary: Regression Mitigation Checklist

PR	Regression	Mitigation	Effort
PR 1	None	N/A	—
PR 2	Proxy/Ollama may not bind port in time on very slow systems	Set generous deadline (10-15s), test on slow hardware	Low
PR 3	test/onboard.test.ts breaks (HEALTH_POLL_COUNT=0)	Add NEMOCLAW_HEALTH_DEADLINE_MS env var; update test	Medium
PR 3	test/cli.test.ts breaks (both env vars)	Update runWithEnv() to use new env var	Low
PR 3	External consumers using HEALTH_POLL env vars	Deprecated alias: convert count×interval to deadline, log warning	Medium
PR 3	"within 60s" error message becomes inaccurate	Parameterize with actual deadline value	Low
PR 4	test/wsl2-probe-timeout.test.ts — entire file	Rewrite: test calibrate() and probe-pool instead	High
PR 4	getValidationProbeCurlArgs export removed	Remove from exports, update any external importers	Low
PR 5	None (additive, backward compatible)	N/A	—
PR 6	Resume message ordering may change under parallelism	Test that resume skip messages still appear correctly	Medium
PR 6	Race conditions if gateway fails while prompts are active	Cancel prompt + clean up on gateway failure signal	Medium
PR 7	None (observability only)	N/A	—

Critical finding: PR 6 (parallel orchestration) has a hidden race condition

If gateway startup fails while the user is in the middle of answering provider selection prompts, the current sequential flow handles this cleanly (gateway failure happens before prompts start). With parallel execution:

• Gateway fails → but user is mid-prompt → need to cancel the prompt and surface the error
• Gateway succeeds slowly → provider selection completes first → inference setup must wait for gateway → need a Promise to coordinate
• User cancels (Ctrl+C) during prompts → need to abort the background gateway start → AbortSignal needed

Recommendation: PR 6 should include explicit cancellation handling via AbortController. The waitUntil framework already supports signal?: AbortSignal for this reason. Gateway failure should propagate as an abort signal to any concurrent operations.

[ksapru]

Hey @ericksoa, I got out a PR to refactor the NemoClaw CLI to replace inefficient and platform-dependent spawnSync("sleep") subprocess calls with a native Node.js wait utility. In the roadmap outlined above, this corresponds to PR1. Low risk.

[cv]

Cross-link audit note: several pieces of this latency plan have already landed or are in-flight, but were easy to miss from the issue thread:

Landed:

• refactor(cli): replace spawnSync("sleep") with native wait utility #2027 — Phase 1-style cleanup: replaced spawnSync("sleep") with a native wait utility.
• refactor(cli): replace race-condition sleeps with readiness probes #2442 — replaced several race-condition sleeps with readiness probes (waitForPort, waitForHttp, waitUntil). The PR body references this issue, but the Markdown link is malformed, so the backlink may not be obvious.
• fix(onboard): retry inference validation on transient upstream failures #3104 — retry inference validation on transient upstream 502/503/504 and curl-timeout failures. This is not the full adaptive-timeout design, but it addresses one validation fragility class discussed here.

Still open / in-flight:

• perf(onboard): add deadline-based gateway wait #2492 — deadline-based gateway wait. This looks like a scoped step for this issue, but may need refresh/rebase because refactor(cli): replace race-condition sleeps with readiness probes #2442 moved wait helpers under src/lib/core/wait.ts.
• perf(cli): add opt-in onboard profiling traces #3070 — opt-in onboard profiling traces via NEMOCLAW_TRACE_FILE / NEMOCLAW_TRACE_DIR.

Suggested bookkeeping: update the checklist in this issue to mark the sleep primitive and race-condition sleep pieces as done, then decide whether #2492 should be rebased or superseded by the newer wait utility structure.