feat: liveness probes, recovery strategies, and skip_if rename (#111)

## Summary

- Adds k8s-inspired **readiness and liveness probes** for both `command`
and `start_server` nodes to detect unhealthy state (dropped SSH tunnels,
crashed processes, unreachable databases)
- Adds **recovery strategies** (`notify`, `restart_node_and_dependents`,
`restart_environment`) configurable at project/node/variant level
- Renames `verify` to **`skip_if`** for clarity (alias preserved)
- Bumps schema version to `"2"` (v1 fully backward compatible)
- Daemon automatically detects liveness failures and triggers recovery
via `veld restart` subprocess

### New config shape

```json
{
  "schemaVersion": "2",
  "recovery_strategy": "restart_node_and_dependents",
  "nodes": {
    "database": {
      "variants": {
        "dblab": {
          "type": "command",
          "command": "./scripts/dblab/start.sh veld-${veld.run}",
          "on_stop": "./scripts/dblab/stop.sh veld-${veld.run}",
          "outputs": ["DATABASE_URL"],
          "skip_if": "./scripts/dblab/verify.sh veld-${veld.run}",
          "probes": {
            "liveness": {
              "type": "command",
              "command": "pg_isready -h ${DB_HOST} -p ${DB_PORT}",
              "interval_ms": 5000,
              "failure_threshold": 3,
              "max_recoveries": 3
            }
          }
        }
      }
    }
  }
}
```

### Key changes

| Area | Changes |
|------|---------|
| Schema | `probes` (readiness + liveness), `recovery_strategy`,
`skip_if`, `LivenessProbe` def |
| Config | `RecoveryStrategy` enum, `ProbesConfig`, `LivenessProbe`,
`resolve_recovery_strategy()`, `readiness_probe()`, `liveness_probe()` |
| State | `NodeStatus::Unhealthy`, `RunStatus::Recovering`, liveness
tracking fields |
| Orchestrator | Uses `probes.readiness` (fallback to `health_check`),
readiness probes for command nodes, `skip_if` rename |
| Daemon | Liveness monitor with per-probe intervals, failure counting,
recovery triggering, `veld restart` subprocess |
| Graph | `get_dependents()` for recovery subgraph identification |
| Recovery | New module: `stop_subgraph()`, state helpers, strategy
resolution |

## Test plan

- [x] All 121 existing + new tests pass (`cargo test`)
- [x] Zero clippy warnings (`cargo clippy`)
- [x] Schema v1 configs still load (backward compat)
- [x] `verify` alias works for `skip_if`
- [x] `probes.readiness` takes precedence over `health_check`
- [x] Recovery strategy resolution: variant > node > global > default
- [x] `get_dependents()` correctly identifies transitive dependents
- [ ] Manual test: command node with liveness probe detects failure and
triggers restart

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: peter-adam-dy

AGENTS.md

@@ -73,9 +73,9 @@ When a change introduces new config fields, CLI flags, subcommands, or user-visi
 |------|----------------|
 | `README.md` | Features list, CLI reference table, Configuration section |
 | `docs/configuration.md` | Config field reference (top-level table, field section, variant table) |
-| `skills/veld-config/SKILL.md` | Agent-facing config reference |
-| `skills/veld-usage/SKILL.md` | Agent-facing CLI reference |
-| `schema/v1/veld.schema.json` | JSON Schema (usually updated in code, but verify) |
+| `skills/veld/SKILL.md` | Agent-facing skill (quick reference, gotchas) |
+| `skills/veld/reference/config.md` | Agent-facing config reference |
+| `schema/v2/veld.schema.json` | JSON Schema for v2 configs (probes, recovery, skip_if) |
 | `website/llms-full.txt` | LLM-facing docs (if applicable, see `website/AGENTS.md`) |
 
 If the change is purely internal (refactor, bugfix with no new surface area), this checklist does not apply.

Cargo.lock

@@ -724,7 +724,7 @@ No cross-compilation until v1 is stable. No Tauri. No GTK. No npm in CI.
         "local": {
           "type": "command",
           "script": "./scripts/clone-db.sh",
-          "verify": "./scripts/verify-db.sh",
+          "skip_if": "./scripts/verify-db.sh",
           "outputs": ["DATABASE_URL"],
           "sensitive_outputs": ["DATABASE_URL"]
         },

PRD.md

@@ -17,7 +17,8 @@ No port numbers. No manual wiring. Just clean, stable, human-readable URLs.
 - **No port numbers** — work with stable HTTPS URLs instead of `localhost:3847`
 - **Dependency graph** — resolves node dependencies, parallelizes startup, reverse-order teardown
 - **TLS by default** — Caddy's internal CA handles TLS termination, auto-trusted during setup
-- **Health checks** — two-phase checks (TCP port + HTTP endpoint) before marking services healthy
+- **Health checks** — readiness probes (two-phase: TCP port + HTTP/command) gate startup; liveness probes detect failures after startup (e.g., dropped SSH tunnels)
+- **Automatic recovery** — when liveness probes detect failure, the environment is automatically restarted (configurable failure threshold and max recovery attempts)
 - **Multiple variants** — same node, different behaviors (local server, Docker, remote URL)
 - **Named runs** — multiple environments coexist; re-running by name is idempotent
 - **Setup / teardown** — project-level lifecycle steps that gate startup (check Docker, create networks) and clean up after stop
@@ -26,6 +27,7 @@ No port numbers. No manual wiring. Just clean, stable, human-readable URLs.
 - **Structured output** — all commands support `--json` for scripting and CI
 - **Browser dashboard** — management UI at `https://veld.localhost` with service health, logs, search, stop/restart
 - **Client-side logs** — captures browser `console.log/warn/error`, exceptions, and promise rejections; view with `veld logs --source client`
+- **Internal logs** — liveness probe outcomes (with stderr), recovery decisions, health state transitions; view with `veld logs --source internal`
 
 ## Install
 
@@ -72,8 +74,8 @@ cargo build --release
 
 ```json
 {
-  "$schema": "https://veld.oss.life.li/schema/v1/veld.schema.json",
-  "schemaVersion": "1",
+  "$schema": "https://veld.oss.life.li/schema/v2/veld.schema.json",
+  "schemaVersion": "2",
   "name": "myproject",
   "url_template": "{service}.{run}.{project}.localhost",
   "nodes": {
@@ -83,7 +85,7 @@ cargo build --release
         "local": {
           "type": "start_server",
           "command": "npm run dev -- --port ${veld.port}",
-          "health_check": { "type": "http", "path": "/health", "timeout_seconds": 30 }
+          "probes": { "readiness": { "type": "http", "path": "/health", "timeout_seconds": 30 } }
         }
       }
     },
@@ -93,7 +95,7 @@ cargo build --release
         "local": {
           "type": "start_server",
           "command": "npm run dev -- --port ${veld.port}",
-          "health_check": { "type": "http", "path": "/", "timeout_seconds": 30 },
+          "probes": { "readiness": { "type": "http", "path": "/", "timeout_seconds": 30 } },
           "depends_on": { "backend": "local" },
           "env": { "NEXT_PUBLIC_API_URL": "${nodes.backend.url}" }
         }
@@ -152,7 +154,7 @@ veld stop --name dev
 ### Step types
 
 - **`start_server`** — long-running process. Veld allocates a port (`${veld.port}`), starts the process, and runs health checks.
-- **`command`** — runs a command to completion. Can emit outputs by writing `key=value` lines to `$VELD_OUTPUT_FILE` (preferred) or via `VELD_OUTPUT key=value` on stdout (legacy, discouraged). Optional `verify` command for idempotency.
+- **`command`** — runs a command to completion. Can emit outputs by writing `key=value` lines to `$VELD_OUTPUT_FILE` (preferred) or via `VELD_OUTPUT key=value` on stdout (legacy, discouraged). Optional `skip_if` command for idempotency.
 
 ### Setup & teardown