feat(fl): per-job fl-server — flip-api scales fl-server to zero between training jobs (NVFLARE)

[garciadias]

Part of #735 (does not close it). Draft — see note below.

What

Phase 0 of #735, rebuilt from scratch on current develop following the #798 review. Decouples fl-api and flip-api from the assumption that the fl-server is always up, behind PER_JOB_FL_SERVER (default false, so flag-off behaviour is byte-identical to today).

The four #798 blockers, fixed:

1. Lazy reconnect re-authenticates. The lazy first-use connect now goes through _reconnect() (fresh Session.__init__ → new AdminAPI), not try_connect() — a failed boot-time connect leaves a half-built cell that AdminAPI.connect()'s if self.cell: return short-circuits.
2. _connected is "currently connected", not "ever connected". _do_command now catches NoConnection, clears _connected, and reconnects before retrying.
3. InternalError no longer escapes. The boot-tolerance path tolerates the InternalError that AdminAPI.login() produces for a server that is up but not ready (the mid-cold-start window).
4. Tolerated set narrowed. Auth/identity failures — AuthenticationError, AuthorizationError, bare FLCommunicationError, and NoConnection("cannot authenticate\…") — still raise even with the flag on. Only transport-down and not-ready are tolerated.

Boot tolerance (session_manager.py): with the flag on, an unreachable server at boot is logged and the session connects lazily on first use; with it off, boot stays fatal exactly as today.

Offline status (check_server_status): returns ServerInfoModel(status="STOPPED") when the server is unreachable, mirroring fl-api-flower, instead of 500-ing the hub's readiness probe.

keep-alive gating (flip-api, C8): when the flag is on and no net is BUSY, the 2-minute keep-alive ping is skipped — no more "no response" noise for servers down by design.

Flag plumbing: PER_JOB_FL_SERVER threaded through fl-api-base, flip-api (unified lenient parsing — empty/commented/"false"/"0"/"no"/"off" = false; "true"/"1"/"yes"/"on" = true, identical in both services), Terraform variables.tf + locals.tf (flip_api and fl_api maps), every NVFLARE compose (dev net-1 + net-2, prod), .env.development.example. Root Makefile no longer exports commented .env lines (which previously exported PER_JOB_FL_SERVER as an empty string).

Why draft

The #798 review's restructure: the tolerate half is unfalsifiable without the scale driver, so this should merge together with (or after) the driver/readiness/IAM work. Phase 1 (TF ignore_changes + scoped ecs:UpdateService) is already up separately as #1141. This PR is the tolerate half, kept as a draft for that reason.

Verification

• fl-services/nvflare/fl-api-base: 197 passed, ruff + mypy clean.
• flip-api (targeted): tests/unit/fl_services/services/test_fl_service.py 94 passed; ruff clean.
• tofu validate (deploy/providers/AWS) clean.

Not in this PR

• The scale driver, STARTING state, readiness gate, and scale-down chokepoint (Phase 2 of feat(fl): per-job fl-server — flip-api scales fl-server to zero between training jobs (NVFLARE) #735).
• Flower backend parity (separate issue).

[codecov]

Codecov Report

❌ Patch coverage is 91.04478% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...s/nvflare/fl-api-base/fl_api/utils/flip_session.py	88.88%	3 Missing ⚠️
flip-api/src/flip_api/config.py	66.66%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!