fix(scheduler): support safe multi-instance scheduler recovery

[rain02333z-spec]

Fixes #4694

Why

Every Gateway process starts ScheduledTaskService. During startup, the scheduler treated every queued or running scheduled-task run in the shared database as a local leftover. In a multi-Pod deployment, a newly started Pod could therefore interrupt a live run owned by another Pod, release uq_scheduled_task_run_active, and allow the next scheduler tick to launch a duplicate execution.

This is especially harmful for overlap_policy=skip: a peer restart can defeat the single-active-run guarantee even though the original run is still executing.

What changed

• The scheduler remains single-instance by default with scheduler.multi_instance: false.
• Operators can explicitly enable lease-aware scheduler recovery across Gateway instances with scheduler.multi_instance: true.
• Multi-instance scheduler startup fails fast unless the deployment uses shared Postgres, run_ownership.heartbeat_enabled: true, and run_events.backend: db.
• Startup and periodic reconciliation preserve scheduled rows backed by a live pending/running run lease and reclaim only terminated/recovered runs or expired leases through an atomic claim_for_takeover() update.
• Scheduled-run and parent-task association recovery uses metadata when post-launch bookkeeping missed both links, while stale parent last_run_id values are ignored.
• Queued pre-launch rows reuse the existing scheduled_tasks.lease_owner / lease_expires_at dispatch lease. Multi-instance manual triggers acquire the same short lease before creating a scheduled run row.
• update_after_launch() fences stale Pods by expected lease owner, so delayed writes cannot overwrite a newer Pod's status or lease.
• Multi-instance due-task claims use a short Postgres transaction advisory lock and count active scheduled rows plus valid dispatch reservations, making max_concurrent_runs a shared global cap.
• Existing Postgres FOR UPDATE SKIP LOCKED claims, the scheduled-run partial unique index, and run ownership leases remain the concurrency primitives; no leader-election service is introduced.
• Configuration version, reload-boundary descriptions, README, backend architecture, and Helm documentation are synchronized.

Surface area

• Frontend UI - no frontend changes
• Backend API - no endpoint or response-shape changes
• Agents / LangGraph - no graph or prompt changes
• Sandbox - no sandbox changes
• Skills - no skill changes
• Dependencies - no dependency changes
• Default behavior change - multi-worker scheduler startup behavior is explicitly gated; multi-instance behavior is opt-in
• Docs / tests / CI only - this PR includes runtime behavior changes

Screenshots / Recording

Not applicable; this change has no frontend UI.

Bug fix verification

• Regression coverage:
  • backend/tests/test_scheduled_task_repository.py: live-peer preservation, expired-run takeover, queued dispatch lease, once-task recovery, metadata fallback, stale owner fencing, and global reservation accounting
  • backend/tests/test_scheduled_task_service.py::test_both_post_launch_association_writes_can_fail_without_releasing_slot
  • backend/tests/test_multi_worker_postgres_gate.py: single-instance gate plus unsafe multi-instance prerequisites
  • backend/tests/test_scheduled_task_postgres.py: cross-Pod advisory-lock budget claim and Postgres metadata/takeover integration cases
• Did the new tests go red on main and green on this branch? Not rerun against main in this checkout. The branch-focused regression suite passes; the original startup-gate regression was verified red before the gate change.

Validation

• cd backend && uv run pytest --basetemp .pytest-tmp-final3 tests/test_multi_worker_postgres_gate.py tests/test_scheduled_task_service.py tests/test_scheduled_task_repository.py tests/test_scheduled_task_dispatch_race.py tests/test_scheduled_task_postgres.py -q - 72 passed, 2 skipped
• cd backend && uv run pytest --basetemp .pytest-tmp-final3 tests/test_reload_boundary.py -q - 10 passed
• cd backend && uv run ruff check . - passed
• cd backend && uv run ruff format --check . - 1094 files already formatted
• git diff --check - passed
• Postgres integration tests were skipped locally because TEST_POSTGRES_URI is unset; CI must provide that variable before treating the Postgres cases as executed.
• A broader local run was 91 passed, 2 skipped, 1 failed; the sole failure is the existing Windows inability to execute scripts/config-upgrade.sh through bash (exit 127), unrelated to scheduler behavior.

AI assistance

Tool(s) used: Codex

How you used it: Codex analyzed the startup recovery path for Issue #4694, implemented the lease-aware multi-instance scheduler fix, added regression tests and documentation, ran the focused and static validation suites, and updated this PR with the verified results.

• I've read and understand every line of this change and take responsibility for it - it's not unreviewed AI output.

[CLAassistant]

All committers have signed the CLA.