fix: surface FL runs that fail after submission (#1001)

[atriaybagur]

Closes #1001.

The failure mode

fl-api submits a job, gets a run-id, writes it to fl_job.fl_backend_job_id and never asks about it again. Everything the hub knows about a run in progress is reported by the run, through the flip package. A run that dies before it can report anything — the canonical case being an exception at module scope in server_app.py, which kills the ServerApp the instant it starts — therefore reports nothing at all:

• the model sits at INITIATED indefinitely,
• fl_logs holds only the queue position,
• the SuperLink logs Started task / Finished task and nothing else,
• the cause is visible only to whoever knows to run flwr log <run-id> local inside the fl-api container.

Where the polling lives, and why

In the hub (flip-api), as a new scheduled sweep — flip_api/fl_services/reconcile_failed_jobs.py, registered on the existing BackgroundScheduler at SCHEDULER_FL_JOB_RECONCILE_RATE (default 1 min).

The issue left this open between a background task in fl-api and the hub's FL scheduler. Reasons for the hub:

1. No new credential surface. Acting on a failure means writing fl_logs and moving the model to ERROR. From fl-api that is an authenticated call to flip-api, and fl-api-flower holds no INTERNAL_SERVICE_KEY / FLIP_API_INTERNAL_URL in any environment today (dev compose, prod compose, ECS task definitions). Only fl-server does. Adding them would put a hub service key in a second container to fix a reporting gap.
2. The hub already owns these transitions. update_model_status(ERROR) is the same call the prepare-failure path uses, and its existing terminal-status branch completes the FLJob and releases the net. Doing that from fl-api would mean duplicating or re-implementing that sequencing across an HTTP boundary.
3. It is stateless and restart-safe. The sweep recomputes from the DB every tick. An in-process poller in fl-api dies with its container and forgets the run it was watching.
4. The hub already polls this exact endpoint. extract_current_job_data calls the net's /list_jobs during aborts; the shared job-metadata contract ([Bug]: Converge the FL-backend job-metadata contract across flip-api, fl-api-flower, fl-api-base #490) already normalises both backends' statuses.

What fl-api-flower gains is the one thing only it can do: GET /run_logs/{run_id}, which runs flwr log <id> local --show and returns a bounded, secret-masked tail.

The hub's lookup (get_backend_job_metadata) returns the whole /list_jobs contract item rather than just the status, so it also picks up status_details — see below — without a second call.

How it behaves

• The sweep selects in-flight jobs only: FLJob.status == IN_PROGRESS, a non-null backend job id, and a model still in INITIATED / PREPARED / RUNNING. Anything already settled is not even polled — which also makes the sweep idempotent, since the ERROR it writes takes the model out of that set.
• Two conditions act, both meaning "this run will never report":
  • the backend lists the job as FAILED;
  • the backend does not list the job at all, and it has been unlisted past FL_JOB_UNLISTED_GRACE_MINUTES (default 30). The SuperLink keeps run state in memory (no --database is passed anywhere in-tree), so a SuperLink restart forgets every run — the same silent death, with the evidence gone too. The feed row says so honestly ("backend restarted, run and log lost"); no log fetch is attempted, since the backend that forgot the run forgot its log with it. The grace period keeps a transient listing hiccup from erroring a healthy run.
• Nothing else acts. FINISHED is deliberately ignored: a run whose ServerApp has finished is routinely still uploading results, and the hub's own RESULTS_UPLOADED callback is the authority there. PENDING/RUNNING/STOPPED are no-ops, and so is UNKNOWN (below).
• The model status is re-read after the (network) calls, as a column select that bypasses the session identity map, so a result or an abort landing mid-poll wins the race.
• The feed row and the ERROR write commit atomically (add_log(..., transaction=session) defers to update_model_status's commit), so a failed status write can never leave an orphan feed row for the next tick to duplicate.
• Everything is per-job best-effort: an unreachable net, a malformed response or a failed write is rolled back and logged, and the sweep moves to the next job.

The UNKNOWN contract value. This PR gives the destructive meaning of FAILED a second look: previously both adapters' normalize_status defaulted unmapped native statuses to FAILED — written for the abort path, where "assume failed" was conservative. The sweep makes that default destructive (model to ERROR, net freed while the real run may still be executing). The shared #490 contract therefore gains a sixth value: an unmapped native status — i.e. one added by a framework upgrade — now surfaces as UNKNOWN, which is a no-op everywhere: the sweep skips it, the abort path already filters to RUNNING, and _find_terminal_run no longer treats an uninterpretable status as proof a run is terminal (an abort that can't be confirmed now 500s loudly instead of no-op'ing). Both maps are pinned exhaustive by tests against the installed frameworks (flwr.common.constant.Status/SubStatus, nvflare.apis.job_def.RunStatus), so UNKNOWN is unreachable until a framework upgrade — but deploy the hub before the FL API images all the same, since an old hub's IJobMetaData would reject it.

Log handling. The tail, not the head: a Flower run log opens with the per-run uv sync over the app's whole dependency set and the traceback is at the far end. fl-api-flower truncates to FLOWER_RUN_LOG_MAX_CHARS (default 8000), advancing the cut to a line boundary and flagging truncated; flip-api caps again at 8000 as belt-and-braces. If the log can't be retrieved the feed row names the manual fallback for the net's backend: flwr log … local --show on a Flower net, docker logs fl-server-net-<n> on an NVFLARE net (see The NVFLARE fallback below — the two more obvious destinations don't work).

Leading with the cause (status_details). The log tail alone buries the reason. Measured on a real
failed run: the stored row is 5,278 bytes, of which ~3,400 is the single Installed: [...] line from the
per-run uv sync, and the rendered <p> is ~2,300px tall — the traceback sits far below the fold of a
fixed-height feed panel. But flwr ls already reports a one-line cause per run, in the very response the
sweep reads for the status:

status:         finished:failed
status-details: ServerApp failed with exception: FLIP1003-VERIFY: injected module-scope failure in models.py

So the shared #490 contract gains an optional status_details, and the feed row leads with
Reported cause: … before the tail. This costs no extra call, no subprocess, no new endpoint. The tail
still follows, because it carries the file and line number that the one-liner does not.

fl-api-flower fills it from status-details, normalising flwr's literal "N/A" (written for a run with
nothing to say) to absent, collapsing whitespace, applying the same redact_secrets masking as the run log,
and bounding it at 500 chars — the text is a researcher-authored exception message from a container holding
a hub service key. NVFLARE leaves it None: nvflare.apis.job_def.JobMetaKey has no error/details key
at all (the nearest, job_deploy_detail, reads ['server: OK', 'Trust_2: OK'] even on a job that then died
with FINISHED:EXECUTION_EXCEPTION). Unlike UNKNOWN, this half of the contract is deploy-order-safe in
both directions: the field is optional with a default, and IJobMetaData already ignores extra fields.

The NVFLARE fallback. Its wording is now docker logs fl-server-net-<n>, because the two destinations
the original hint named are both dead ends — verified live, not assumed. POST /<job-id>/show_errors/server
returns HTTP 500 against the installed NVFLARE (FLIP_Session._do_command() got an unexpected keyword argument 'enforce_meta'), filed as #1032. And the job workspace holds opaque data / meta /
workspace blobs, not a readable log: a filesystem-wide grep -rl for the failing exception string across
a failed run's fl-server matched only the uploaded app bundle, never the traceback. The container's stdout
is where it actually lives.

Rendering. Timeline.vue now renders failure rows with whitespace-pre-wrap (success rows keep the default), so the stored traceback keeps its line structure in the activity feed — the difference between meeting the "readable without shelling into a container" criterion and technically storing the bytes.

Secrets. The issue's brief mentioned an existing scrub_args helper in fl_api/app.py — there is no such helper anywhere in the repo (grep -rn scrub_args is empty), so I wrote one: fl_api/utils/redaction.py. A run log is whatever researcher-supplied ServerApp code wrote to stdout/stderr, in a container that holds a hub service key, so it is not trusted to be secret-free. It masks SigV4 query parameters on presigned URLs (individually, so the object path stays readable), …key|token|secret|password|credential followed by :/=, and bare AKIA/ASIA access key ids. Redaction runs before truncation, so a cut landing inside a key=<secret> pair can't strip the keyword the matcher needs. It is biased towards over-redaction and is damage limitation, not a guarantee — a secret printed with no recognisable keyword still gets through.

What is and isn't covered

Covered: the ServerApp. A run that fails at any point after a successful submit — including a backend restart that loses the run — on either backend (see below).

Not covered: a ClientApp that dies at a trust. Its logs live on that trust's SuperNode, which the hub cannot read — out of scope per the issue, worth a follow-up.

Both backends, deliberately — but the exposure differs. The sweep talks only to the shared /list_jobs contract, so it runs for NVFLARE nets too. On Flower the gap is wide open: the template server_app.py has no top-level exception reporting, so most failures say nothing. On NVFLARE the gap is narrower — flip_server_event_handler already reports FATAL_SYSTEM_ERROR and END_RUN terminal statuses — and what the sweep adds there is the jobs that never reach END_RUN (FINISHED:CAN_NOT_SCHEDULE, FAILED_TO_RUN, ABANDONED, a server crash, a failure before the handler is built). The remaining asymmetry is diagnostic, not behavioural. The sweep itself never branches on backend — same poll, same FAILED/unlisted conditions, same ERROR, same net release. What differs is how much a row can say: /run_logs exists only on fl-api-flower, and status_details only Flower populates. fetch_run_logs degrades to None on any failure (including the 404 an NVFLARE net returns), so an NVFLARE failure surfaces status-only, and picks up logs for free if fl-api-base ever grows the endpoint. Both gaps are properties of what each framework exposes, not of this design.

Follow-ups (separate PRs/issues, deliberately not here):

• Wrapping the Flower templates' @app.main() in try/except → send_handled_exception + ERROR — the in-run half of this fix, higher-fidelity for in-main() failures, and its own review.
• Persisting SuperLink run state (--database) across dev/secure compose and the ECS task defs — an operational change; the unlisted-grace fallback here removes the urgency.
• fl-api-base show_errors/show_stats/reset_errors 500: FLIP_Session._do_command narrows the base signature #1032 — show_errors 500s on fl-api-base (NVFLARE API drift). Pre-existing; this PR routes around it rather than depending on it.
• Stored FL run log keeps raw ANSI escapes, rendering as junk in the activity feed #1033 — the stored run log keeps raw ANSI escapes, so every log line renders as [92mINFO [0m: junk in the feed. Found by eyeballing a real row in the browser (below). Deliberately not fixed here to keep this diff to one concern.

Verified — unit / static

• make -C fl-services/flower/fl-api-flower local_test — ruff + mypy clean, 139 passed, 0 failed (test_run_logs.py, test_redaction.py, the flwr status-map exhaustiveness pin, and 4 new status_details cases: the happy path, flwr's "N/A" sentinel, collapse+redact+bound, and an absent key on an older flwr).
• make -C fl-services/nvflare/fl-api-base local_test — ruff + mypy clean, 166 passed, 0 failed.
• flip-api: ruff check . clean, mypy . clean (364 files), pytest tests/unit — 1559 passed, 0 failed; pytest tests/integration — 127 passed, 0 failed, of which 3 are new (test_fl_reconcile_db_flow.py: the in-flight join, the settled-job filter, and the unlisted-past-grace resolution, all against the throwaway Testcontainers Postgres — the join is exactly the thing a mocked session can't catch going silently wrong).
• The contract field set is pinned in three places that must move together — IJobMetaData plus both adapters' JobMetadata — so a field added on one side only fails a test rather than going quietly missing.
• flip-ui: npm run lint (0 errors; 3 pre-existing max-len warnings in EditProjectDrawer.vue, untouched here), npm run test:unit — 1250 passed (Timeline pre-wrap spec added test-first).
• scripts/check_fl_api_validation_sync.sh still passes.
• detect-secrets scan --baseline .secrets.baseline over the new files: no new findings.
• flwr 1.32 source read to confirm the mechanics behind --show: flwr/cli/log.py::print_logs breaks after the first streamed message, and the SuperLink's StreamLogs servicer yields the whole accumulated log since after_timestamp=0 in a single message and ends the stream once the run is FINISHED (which includes finished:failed). Also confirmed the SuperLink's --database default is the in-memory sentinel (flwr/server/app.py), which is what makes the unlisted-run fallback necessary.

Verified — live on the dev stack

The four gaps the earlier revision of this description listed as unverified have now been driven on a real stack (hub on this branch, both FL backends in turn, project reused so DICOM was pulled once). The fault injected on both backends is the canonical #1001 shape: a module-scope raise ImportError in the researcher's own models.py.

Flower — end to end. Run 8621698533979188955 reached finished:failed; the sweep resolved it within a tick — model INITIATED → ERROR, FLJob → COMPLETED, scheduler back to AVAILABLE. GET /run_logs talked to a real SuperLink for the first time and returned the genuine traceback, ending:

File ".../app/server_app.py", line 30, in <module>
    from app.models import get_model
File ".../app/models.py", line 40, in <module>
    raise ImportError("FLIP1003-VERIFY: injected module-scope failure in models.py")
ImportError: FLIP1003-VERIFY: injected module-scope failure in models.py
ERROR:     Exit Code: 607

NVFLARE — end to end, and it earns its keep. Job 55c8e4d8-… reached FINISHED:EXECUTION_EXCEPTION → FAILED; the sweep errored the model and freed the net in ~60s. Worth noting why nothing else reported it: the server config builds persistor from models.get_model at components.#1, and flip_server_event_handler is components.#5 — so the job died at component construction before the reporting handler existed. This is precisely the class of NVFLARE failure the sweep exists for. fetch_run_logs degraded exactly as designed against the live NVFLARE FL API (404 → None → status-only row plus the backend hint).

A healthy run is untouched — checked, not assumed. An unmodified NVFLARE run went INITIATED → PREPARED → RUNNING with the sweep polling it across several ticks and taking no action. When it later failed for an unrelated reason (a trust-side data-mount mismatch in my test environment), NVFLARE's own client reporter handled it — no sweep row, no duplicate, no interference.

status_details, live. Confirmed on the rebuilt flower-fl-api:dev against a real SuperLink: a running run reports status-details: "N/A" (normalised to absent, so no Reported cause: N/A in the feed), and a failed one reports the ServerApp's exception verbatim, which the feed row now leads with.

Timeline rendering, in a browser. The live route is blocked by a deliberate vite.config.mts guard refusing VITE_E2E=true in development mode, so this ran through the docs harness with the real stored bytes as the fixture. whitespace-pre-wrap applies to the failure row and not the success row, and the traceback keeps its line structure. It also surfaced two things only a browser shows: the ANSI junk now filed as #1033, and the sheer size of the row (~2,300px, mostly dependency list) — which is what motivated status_details above.

Still not verified

• A fully successful training run through the sweep. The healthy-run control above proves the sweep leaves a live RUNNING job alone and defers to the run's own reporting, but that particular run ended in an unrelated environment failure rather than RESULTS_UPLOADED.
• The unlisted-past-grace path live. Covered by unit and integration tests; not driven by actually restarting a SuperLink under an in-flight job.
• Related, from reading rather than running: the unlisted grace is anchored on FLJob.started, not on when the run was first seen missing — so a job older than the window has no protection from a single transient omission. Low practical risk (both adapters raise rather than return a short list when the backend is unreachable, so a 200 without the run really does mean the backend lost it), but the docstring's "keeps a transient listing hiccup from erroring a healthy run" only holds for jobs younger than the grace period. Flagged rather than changed — adding first-unlisted state to make it exact is a bigger call than this PR should take.

Also worth knowing

• fl-services/flower/fl-api-flower/fl_api/utils/upload.py carries no Apache 2.0 header, contrary to the repo rule. Untouched here — separate fix.

Acceptance Criteria

Imported from issue #1001

• A Flower run that fails after submission is reported to the Central Hub rather than leaving the
  model at INITIATED. The model reaches a terminal state and the failure is visible in the UI.
• The ServerApp's output for the failed run is captured into fl_logs (which already carries a
  success boolean), so the cause is readable without shelling into a container.
• A run that succeeds is unaffected — no change to timing, status transitions or metrics.
• Covered by tests in fl-services/flower/fl-api-flower/tests/ alongside the existing
  test_submit_run.py / test_check_status.py.
• The flwr log <run-id> <superlink> command is documented as the manual fallback.

[github-actions]

✅ Acceptance criteria have been automatically imported from the linked issue(s) and added to the PR description.

[github-actions]

✅ Acceptance criteria have been automatically imported from the linked issue(s) and added to the PR description.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!