feat!: liveness probes, recovery strategies, and skip_if rename

Add k8s-inspired readiness and liveness probes to detect unhealthy nodes
(e.g., dropped SSH tunnels, crashed processes) and automatically recover.

- New `probes` field on variants with `readiness` (replaces `health_check`)
  and `liveness` (runs continuously after node becomes healthy)
- New `recovery_strategy` field at project/node/variant level:
  `notify`, `restart_node_and_dependents` (default), `restart_environment`
- Rename `verify` to `skip_if` (alias preserved for backward compat)
- Schema version bump to "2" (v1 still accepted)
- New `Unhealthy` node status and `Recovering` run status
- Daemon liveness monitor with per-probe interval, failure threshold,
  max recoveries, and automatic restart via `veld restart` subprocess
- New `get_dependents()` graph utility for recovery subgraph identification
- Recovery module for stopping/restarting node subgraphs

BREAKING CHANGE: Schema version bumped to "2". The `health_check` field
on start_server variants is deprecated in favor of `probes.readiness`.
The `verify` field on command variants is renamed to `skip_if`. Both old
field names are still accepted as aliases for backward compatibility, but
new configs should use the new names. The `probes` field and
`recovery_strategy` field are new additions available on all variant types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: peter-adam-dy

AGENTS.md

@@ -73,9 +73,9 @@ When a change introduces new config fields, CLI flags, subcommands, or user-visi
 |------|----------------|
 | `README.md` | Features list, CLI reference table, Configuration section |
 | `docs/configuration.md` | Config field reference (top-level table, field section, variant table) |
-| `skills/veld-config/SKILL.md` | Agent-facing config reference |
-| `skills/veld-usage/SKILL.md` | Agent-facing CLI reference |
-| `schema/v1/veld.schema.json` | JSON Schema (usually updated in code, but verify) |
+| `skills/veld/SKILL.md` | Agent-facing skill (quick reference, gotchas) |
+| `skills/veld/reference/config.md` | Agent-facing config reference |
+| `schema/v2/veld.schema.json` | JSON Schema for v2 configs (probes, recovery, skip_if) |
 | `website/llms-full.txt` | LLM-facing docs (if applicable, see `website/AGENTS.md`) |
 
 If the change is purely internal (refactor, bugfix with no new surface area), this checklist does not apply.

Cargo.lock

@@ -724,7 +724,7 @@ No cross-compilation until v1 is stable. No Tauri. No GTK. No npm in CI.
         "local": {
           "type": "command",
           "script": "./scripts/clone-db.sh",
-          "verify": "./scripts/verify-db.sh",
+          "skip_if": "./scripts/verify-db.sh",
           "outputs": ["DATABASE_URL"],
           "sensitive_outputs": ["DATABASE_URL"]
         },

PRD.md

@@ -17,7 +17,8 @@ No port numbers. No manual wiring. Just clean, stable, human-readable URLs.
 - **No port numbers** — work with stable HTTPS URLs instead of `localhost:3847`
 - **Dependency graph** — resolves node dependencies, parallelizes startup, reverse-order teardown
 - **TLS by default** — Caddy's internal CA handles TLS termination, auto-trusted during setup
-- **Health checks** — two-phase checks (TCP port + HTTP endpoint) before marking services healthy
+- **Health checks** — readiness probes (two-phase: TCP port + HTTP/command) gate startup; liveness probes detect failures after startup (e.g., dropped SSH tunnels)
+- **Automatic recovery** — configurable recovery strategies (`notify`, `restart_node_and_dependents`, `restart_environment`) when liveness probes fail
 - **Multiple variants** — same node, different behaviors (local server, Docker, remote URL)
 - **Named runs** — multiple environments coexist; re-running by name is idempotent
 - **Setup / teardown** — project-level lifecycle steps that gate startup (check Docker, create networks) and clean up after stop
@@ -26,6 +27,7 @@ No port numbers. No manual wiring. Just clean, stable, human-readable URLs.
 - **Structured output** — all commands support `--json` for scripting and CI
 - **Browser dashboard** — management UI at `https://veld.localhost` with service health, logs, search, stop/restart
 - **Client-side logs** — captures browser `console.log/warn/error`, exceptions, and promise rejections; view with `veld logs --source client`
+- **Internal logs** — liveness probe outcomes (with stderr), recovery decisions, health state transitions; view with `veld logs --source internal`
 
 ## Install
 
@@ -72,8 +74,8 @@ cargo build --release
 
 ```json
 {
-  "$schema": "https://veld.oss.life.li/schema/v1/veld.schema.json",
-  "schemaVersion": "1",
+  "$schema": "https://veld.oss.life.li/schema/v2/veld.schema.json",
+  "schemaVersion": "2",
   "name": "myproject",
   "url_template": "{service}.{run}.{project}.localhost",
   "nodes": {
@@ -83,7 +85,7 @@ cargo build --release
         "local": {
           "type": "start_server",
           "command": "npm run dev -- --port ${veld.port}",
-          "health_check": { "type": "http", "path": "/health", "timeout_seconds": 30 }
+          "probes": { "readiness": { "type": "http", "path": "/health", "timeout_seconds": 30 } }
         }
       }
     },
@@ -93,7 +95,7 @@ cargo build --release
         "local": {
           "type": "start_server",
           "command": "npm run dev -- --port ${veld.port}",
-          "health_check": { "type": "http", "path": "/", "timeout_seconds": 30 },
+          "probes": { "readiness": { "type": "http", "path": "/", "timeout_seconds": 30 } },
           "depends_on": { "backend": "local" },
           "env": { "NEXT_PUBLIC_API_URL": "${nodes.backend.url}" }
         }
@@ -152,7 +154,7 @@ veld stop --name dev
 ### Step types
 
 - **`start_server`** — long-running process. Veld allocates a port (`${veld.port}`), starts the process, and runs health checks.
-- **`command`** — runs a command to completion. Can emit outputs by writing `key=value` lines to `$VELD_OUTPUT_FILE` (preferred) or via `VELD_OUTPUT key=value` on stdout (legacy, discouraged). Optional `verify` command for idempotency.
+- **`command`** — runs a command to completion. Can emit outputs by writing `key=value` lines to `$VELD_OUTPUT_FILE` (preferred) or via `VELD_OUTPUT key=value` on stdout (legacy, discouraged). Optional `skip_if` command for idempotency.
 
 ### Setup & teardown