fix: surface FL runs that fail after submission (#1001)

.env.development.example

@@ -341,6 +341,11 @@ FL_APP_DESTINATION_BUCKET=s3://${FLIP_APP_BUNDLES_BUCKET_NAME}/app_destinations
 # BANDIT_TIMEOUT_SECONDS=60
 # How often the sweep re-checks uploads left SCANNING by an app restart.
 # SCHEDULER_MALWARE_SCAN_RECONCILE_RATE=1
+# How often the hub asks each net's FL API whether an in-flight job has failed
+# (FLIP#1001), and how long a job may go unlisted by its backend (a SuperLink
+# restart loses its in-memory run state) before it is treated as dead.
+# SCHEDULER_FL_JOB_RECONCILE_RATE=1
+# FL_JOB_UNLISTED_GRACE_MINUTES=30
 
 # ── FL Net Endpoints ──────────────────────────────────────────────────────
 # Maps FL network IDs to internal API URLs for the FL API containers on the

AGENTS.md

@@ -395,6 +395,16 @@ After changes, evaluate if docs need updating:
 - `PICKLESCAN_FILE_SUFFIXES` / `PICKLESCAN_TIMEOUT_SECONDS` — which uploads get a structural picklescan before promotion (default `.pt .pth .pkl .pickle`) and the wall-clock cap per scan (default 120s). Dangerous globals mark the file `INFECTED` and delete the object; a scan that errors or times out fails closed to `ERROR`. Signature-based AV (GuardDuty Malware Protection for S3) is tracked separately in FLIP#838.
 - `BANDIT_TIMEOUT_SECONDS` — wall-clock cap (default 60s) on the non-blocking Bandit pass over `.py` uploads (FLIP#877, GHSA-8465). Unlike `PICKLESCAN_TIMEOUT_SECONDS` a timeout here just means no findings are recorded (fail-open, never `ERROR`) — Bandit is advisory only and never gates promotion. Findings land on `UploadedFiles.bandit_findings` (`[]` = scanned clean, `NULL` = never scanned) and surface in the UI as an amber indicator. `bandit` is a dependency baked into the `flip-api` image, not the mounted `src/`: under the dev pull-by-default sourcing above, running without `BUILD=true` after picking up this dependency serves an image with no `bandit` binary, so every upload silently records `NULL` findings while the UI and docs still claim a scan ran.
 - `SCHEDULER_MALWARE_SCAN_RECONCILE_RATE` — how often (minutes, default 1) the sweep re-checks uploads left `SCANNING` by an app restart mid-scan.
+- `SCHEDULER_FL_JOB_RECONCILE_RATE` / `FL_JOB_UNLISTED_GRACE_MINUTES` / `FLOWER_RUN_LOG_MAX_CHARS` — how often
+  (minutes, default 1) the hub asks each net's FL API whether an in-flight job has failed, how long (default 30 min)
+  a job may go unlisted by its backend before it is treated as dead (SuperLink run state is in-memory, so a restart
+  forgets every run), and the cap (default 8000 chars) on the run-log tail `fl-api-flower`'s `GET /run_logs/{run_id}`
+  returns. Nothing reports a run that dies *after* submission (an ImportError at ServerApp module scope, say) — the
+  model would otherwise sit at `INITIATED` forever — so the sweep (`flip_api/fl_services/reconcile_failed_jobs.py`)
+  polls for it, moves the model to `ERROR` and stores the log tail on the activity feed. Only `FAILED` and
+  unlisted-past-grace act; `FINISHED` is left to the run's own `RESULTS_UPLOADED` callback, and `UNKNOWN` (an
+  unmapped native status — the shared #490 contract's sixth value, never guessed into FAILED) is a no-op. Covers
+  the ServerApp only — a ClientApp dying at a trust logs to that trust's SuperNode (FLIP#1001).
 
 ## Deployment Architecture
 

CLAUDE.md

@@ -395,6 +395,16 @@ After changes, evaluate if docs need updating:
 - `PICKLESCAN_FILE_SUFFIXES` / `PICKLESCAN_TIMEOUT_SECONDS` — which uploads get a structural picklescan before promotion (default `.pt .pth .pkl .pickle`) and the wall-clock cap per scan (default 120s). Dangerous globals mark the file `INFECTED` and delete the object; a scan that errors or times out fails closed to `ERROR`. Signature-based AV (GuardDuty Malware Protection for S3) is tracked separately in FLIP#838.
 - `BANDIT_TIMEOUT_SECONDS` — wall-clock cap (default 60s) on the non-blocking Bandit pass over `.py` uploads (FLIP#877, GHSA-8465). Unlike `PICKLESCAN_TIMEOUT_SECONDS` a timeout here just means no findings are recorded (fail-open, never `ERROR`) — Bandit is advisory only and never gates promotion. Findings land on `UploadedFiles.bandit_findings` (`[]` = scanned clean, `NULL` = never scanned) and surface in the UI as an amber indicator. `bandit` is a dependency baked into the `flip-api` image, not the mounted `src/`: under the dev pull-by-default sourcing above, running without `BUILD=true` after picking up this dependency serves an image with no `bandit` binary, so every upload silently records `NULL` findings while the UI and docs still claim a scan ran.
 - `SCHEDULER_MALWARE_SCAN_RECONCILE_RATE` — how often (minutes, default 1) the sweep re-checks uploads left `SCANNING` by an app restart mid-scan.
+- `SCHEDULER_FL_JOB_RECONCILE_RATE` / `FL_JOB_UNLISTED_GRACE_MINUTES` / `FLOWER_RUN_LOG_MAX_CHARS` — how often
+  (minutes, default 1) the hub asks each net's FL API whether an in-flight job has failed, how long (default 30 min)
+  a job may go unlisted by its backend before it is treated as dead (SuperLink run state is in-memory, so a restart
+  forgets every run), and the cap (default 8000 chars) on the run-log tail `fl-api-flower`'s `GET /run_logs/{run_id}`
+  returns. Nothing reports a run that dies *after* submission (an ImportError at ServerApp module scope, say) — the
+  model would otherwise sit at `INITIATED` forever — so the sweep (`flip_api/fl_services/reconcile_failed_jobs.py`)
+  polls for it, moves the model to `ERROR` and stores the log tail on the activity feed. Only `FAILED` and
+  unlisted-past-grace act; `FINISHED` is left to the run's own `RESULTS_UPLOADED` callback, and `UNKNOWN` (an
+  unmapped native status — the shared #490 contract's sixth value, never guessed into FAILED) is a no-op. Covers
+  the ServerApp only — a ClientApp dying at a trust logs to that trust's SuperNode (FLIP#1001).
 
 ## Deployment Architecture
 

docs/source/components/component-fl-nodes.rst

@@ -293,6 +293,53 @@ Note the reported upload sizes measure slightly different things per backend —
 The server will also use the package to update the status, as well as to upload the final results, which will be first saved in the server, to the final S3 buckets users can download from.
 
 
+When a run fails after submission
+---------------------------------
+
+Almost everything the Central Hub knows about a run in progress is reported *by* the run,
+through the ``flip`` package above. A run that dies before it can report anything — the
+classic case is an exception at import time in ``server_app.py``, which kills the ServerApp
+the instant it starts — would therefore say nothing at all, leaving the model showing
+``INITIATED`` indefinitely.
+
+The hub closes that gap by polling. Once a minute it asks each net's FL API for the status of
+every job it still believes is in flight; a job the backend reports as failed moves the model
+to ``ERROR``, frees the net, and writes the tail of the run's own log to the model's activity
+feed, where the researcher reads it alongside the round events. A job the backend no longer
+lists at all is treated the same way once it has gone unlisted for a grace period
+(``FL_JOB_UNLISTED_GRACE_MINUTES``, default 30) — the Flower SuperLink keeps run state in
+memory, so a restart forgets every run, and such a run can never report. Nothing else is
+acted on: a run that is pending, running or finished is left entirely to its own reporting.
+
+The feed row leads with the backend's own one-line explanation of the failure, when it has
+one. Flower reports this on every ``flwr ls`` entry, so the hub gets the ServerApp's exception
+type and message from the same call it already makes to read the status — no extra request.
+NVFLARE has no equivalent field, so its rows carry the status alone.
+
+Two things are worth knowing about the captured log. It is a **tail**: a Flower run log opens
+with the per-run dependency install and the cause of a failure is at the far end, so the head
+is dropped. And it is best-effort — if the log cannot be retrieved, the feed says so and
+names the manual fallback. On a Flower net that is to run, inside the net's FL API container:
+
+.. code-block:: bash
+
+   flwr log <run-id> local --show
+
+On an NVFLARE net (whose FL API serves no run-log endpoint today, so its failures always
+surface status-only) the fallback is the fl-server container's own output:
+
+.. code-block:: bash
+
+   docker logs fl-server-net-<n>
+
+NVFLARE keeps no readable per-job log in the job workspace — the workspace holds opaque
+archive blobs — so the container output is where the traceback actually lives.
+
+This covers the **ServerApp**. A ClientApp that dies at a trust writes to that trust's
+SuperNode, which the Central Hub cannot read; such failures still surface only through
+whatever the app itself reports before dying.
+
+
 Privacy filters on shared model updates
 ---------------------------------------
 

fl-services/flower/README.md

@@ -150,3 +150,25 @@ make -C fl-services/flower down
 The central-hub multi-net Flower topology is the separate
 [`deploy/compose.development.flower.yml`](../../deploy/compose.development.flower.yml), driven by the
 root `make up FL_BACKEND=flower`.
+
+## Diagnosing a run that fails after submission
+
+A Flower run whose ServerApp dies logs almost nothing where you would look for it: the SuperLink
+reports `Started task` and `Finished task` for the run in the same breath and says no more, and no
+container's `docker logs` carries the traceback. The run's own log lives in the SuperLink, reachable
+only through the Control API.
+
+The Central Hub polls for this (FLIP#1001): a job the FL API reports as failed drives the model to
+`ERROR` and the tail of the run log is written to the model's activity feed, so the usual first stop
+is the model page rather than a shell. To read the full stream by hand — or to inspect a run
+submitted outside the hub, e.g. via `make submit` — exec into the net's FL API container:
+
+```bash
+docker exec -it flip-fl-api-net-1 uvx flwr log <run-id> local --show     # hub multi-net stack
+docker compose -f fl-services/flower/compose.dev.yml exec fl-api \
+    uvx flwr log <run-id> local --show                                   # standalone dev stack
+```
+
+`--show` prints the stored log and exits; the `flwr log` default (`--stream`) follows it forever.
+Run ids come from `uvx flwr list local` in the same container, or from the `run-id` the submit
+returned. A ServerApp that died at import time ends with a traceback and `ERROR: Exit Code: 607`.

fl-services/flower/fl-api-flower/README.md

@@ -27,6 +27,7 @@ Standalone FastAPI service for Flower deployment runtime.
 - `POST /submit_run/{job_folder}` — submit a previously uploaded application; `job_folder` is the Central Hub `model_id` (UUID). flip-api's production path (also exposed as the hidden `/submit_job` alias)
 - `POST /submit_tutorial/{tutorial_name}` — submit a pre-baked tutorial folder by name (e.g. `numpy`, `xray_classification`); the local tutorial harness targets this
 - `DELETE /abort_run/{run_id}`
+- `GET /run_logs/{run_id}` — a bounded, secret-masked tail of a run's ServerApp log; the Central Hub reads it when it finds a run in a failed state (FLIP#1001)
 
 ## API docs
 
@@ -63,6 +64,27 @@ uvx flwr stop <run_id> local --format json
 
 It returns the full JSON payload from Flower.
 
+The run-logs endpoint runs:
+
+```bash
+uvx flwr log <run_id> local --show
+```
+
+`--show` (rather than the `flwr log` default `--stream`, which follows the log forever) prints what
+the SuperLink has stored for the run and exits. The response is
+`{"run_id": ..., "log": ..., "truncated": ...}`, where `log` is the **last** `FLOWER_RUN_LOG_MAX_CHARS`
+characters (default 8000) of the output: a Flower run log opens with the per-run dependency install and
+the cause of a failure is at the other end, so the head is the half worth dropping. Credential-shaped
+substrings are masked first — a run log is whatever researcher-supplied ServerApp code printed, in a
+container that holds a hub service key, so it is not trusted to be secret-free.
+
+To read the full stream by hand instead, exec into this container and run the same command without
+the truncation:
+
+```bash
+docker exec -it flip-fl-api-net-1 uvx flwr log <run_id> local --show
+```
+
 The server status endpoint checks the Flower SuperLink health service configured by
 `SUPERLINK_HEALTH_ADDRESS` and returns:
 
@@ -87,6 +109,8 @@ If `targets` are omitted, all registered trust names are returned.
 Set these environment variables in the FL API container:
 
 - `SUPERLINK_HEALTH_ADDRESS` (example: `superlink:9097`) — for server status checks
+- `FLOWER_RUN_LOG_MAX_CHARS` (optional, default `8000`) — cap on the run-log tail returned by
+  `/run_logs/{run_id}`. An unset, empty or unparseable value falls back to the default.
 
 ## Development startup with Docker Compose