FLIP-PT-004: authenticated AES-GCM payload encryption + per-trust keys

docs/aes-payload-keys.md

@@ -0,0 +1,146 @@
+# Cross-trust payload encryption — keys and rotation
+
+Payloads exchanged between the hub and trusts (task payloads carrying cohort SQL,
+`encrypted_project_id`, XNAT credentials) are encrypted with **AES-256-GCM** by
+`utils/encryption.py` in `flip-api`, `trust-api`, `imaging-api` and
+`data-access-api` (FLIP-PT-004).
+
+GCM is authenticated: a tampered ciphertext raises `InvalidTag` instead of
+decrypting to attacker-influenced bytes, which the previous AES-CBC scheme
+allowed. The envelope's key-id is bound into the authentication tag, so it cannot
+be relabelled either.
+
+## Envelope
+
+Base64 of `{"v": 1, "kid": ..., "iv": ..., "ct": ...}` — `iv` is a 12-byte GCM
+nonce, `ct` is ciphertext‖tag. The algorithm is fixed rather than negotiated
+(no algorithm-confusion surface); `v` exists so the envelope can be changed later.
+
+## Keys
+
+| Variable | Where | Purpose |
+|---|---|---|
+| `AES_KEY_BASE64` | hub + all trusts | Shared key, registered under kid `shared`. Used when nothing else is set. |
+| `TRUST_AES_KID` + `TRUST_AES_KEY_BASE64` | one trust | That trust's own key. When set, it is used for everything that service encrypts. |
+| `AES_TRUST_KEYS` | hub | JSON `{kid: base64_key}` of per-trust keys, so the hub can encrypt to a specific trust. |
+
+`register_trust` mints a per-trust key and kid (`trust-<trust_id>`) at
+registration, returned in the kit as `trust_aes_key` / `trust_aes_kid`.
+
+## Giving a trust its own key
+
+Per-trust keys contain the blast radius: with one shared key, a single
+compromised trust yields the key for the whole federation.
+
+Registration delivers the key to the trust but leaves it **inert**: the kit gets
+`TRUST_AES_KEY_BASE64` live and `TRUST_AES_KID` commented out. Nothing is stored
+hub-side — the hub has no column for it and cannot re-emit it — so the value shown
+at registration is the only copy, and `AES_TRUST_KEYS` is populated by hand.
+
+**The order matters, and there is no safe partial state.** `TRUST_AES_KID` is the
+switch: setting it live makes the trust encrypt under its own key *and* is what lets
+the trust decrypt anything the hub encrypts to it. Enable the two halves out of step
+and one direction breaks:
+
+* **Trust first, hub not yet:** imaging-api encrypts the new XNAT user's password
+  under the trust's kid, the hub cannot resolve it, and the credentials email is
+  never sent. The failure is swallowed by a broad handler in
+  `private_services/imaging_notifications.py`, so the only symptom is a log line.
+* **Hub first, trust not yet:** `kid_for_trust()` starts selecting the trust's kid
+  for task payloads, but the trust has not loaded that kid, so its poller cannot
+  decrypt them.
+
+So enable both sides and restart both, in one window:
+
+1. Put the trust's key in the hub's `AES_TRUST_KEYS` under `trust-<trust_id>`
+   (source it from a secret store — **not** the application database, or a DB
+   compromise becomes a federation-wide key compromise), and restart the hub:
+   `_keyring()` is memoised for the process lifetime.
+2. Uncomment `TRUST_AES_KID` in that trust's kit (`trust/.env.<CODE>.<env>`) and
+   restart the trust stack.
+
+Re-registering a trust never re-comments a kid an operator has enabled.
+
+`kid_for_trust()` then selects that key automatically; trusts without one keep
+using the shared key, so trusts can be moved over one at a time.
+
+### What a per-trust key does and does not cover
+
+Hub → trust payloads are per-trust keyed only where one ciphertext has exactly one
+reader. That is the task-dispatch path (`GET /tasks/pending`, whose whole batch
+belongs to the polling trust) and the cohort-query submission, which encrypts the
+project id once per trust.
+
+The FL training payload is **deliberately left on the shared key**:
+`fl_service.start_training` hands a single ciphertext to the FL server, which fans it
+out to every participating client. Encrypting under one trust's key would make it
+undecryptable for the rest. Narrowing it needs the FL server to carry a per-client
+payload — a protocol change, not a different `kid` at the call site.
+
+Trust-internal encryption (imaging-api → data-access-api) needs nothing special: on a
+trust, `_default_kid()` already resolves to that trust's own `TRUST_AES_KID`.
+
+### Getting only half of it configured
+
+The two halves are provisioned separately and nothing joins them up, so a partial
+rollout fails asymmetrically:
+
+* **hub → trust degrades silently.** `kid_for_trust()` does not find the trust's kid
+  in the hub's keyring and falls back to the shared key. The trust decrypts it fine.
+  Everything works, without the isolation you configured, and nothing says so.
+* **trust → hub fails loudly**, with `KeyError: No key registered for kid
+  'trust-<uuid>'`.
+
+Because the quiet direction is the dangerous one, flip-api audits the keyring at
+startup (`trusts_services/services/trust_key_config.py`) and logs per-trust key
+coverage. Two operator errors are logged at error level: a kid in `AES_TRUST_KEYS`
+matching no registered trust (a typo or a deleted trust — the key is loaded but never
+selected, which is indistinguishable at runtime from "no key yet"), and a key that is
+not a valid AES length. A malformed `AES_TRUST_KEYS` now fails at boot rather than on
+the first request that encrypts.
+
+`_keyring()` is memoised for the process lifetime, so adding a key needs a restart.
+
+Both registration paths now deliver step 2 automatically: `register-trust
+KIT=<CODE>` writes the two variables into the kit file (they are credential keys
+in `scripts/trust_kit_lib.py`, so they are written once on a new registration and
+never clobbered on the idempotent skip path), and the admin UI's kit modal
+includes them in its copy-all block.
+
+> Step 1 is still manual. The hub does not persist the AES key anywhere — a
+> symmetric key cannot be reduced to a hash the way the api key can — so the value
+> shown at registration is the only copy. Record it into `AES_TRUST_KEYS` at the
+> same time, or the trust has to be re-registered to obtain another.
+
+## Rolling this out to an existing deployment
+
+`decrypt()` also accepts the previous **AES-CBC** format, so a hub and a trust on
+different builds still understand each other while hosts are upgraded — which
+matters for on-prem trusts that cannot all be restarted at once. `encrypt()` is
+GCM-only, so no new unauthenticated ciphertext is produced, and nothing is stored
+encrypted at rest (task payloads are held as plaintext in the hub database and
+encrypted at dispatch), so there is no data to migrate.
+
+There is also no key change: `AES_KEY_BASE64` stays as it is, and per-trust keys
+are inert until provisioned. A trust operator changes no configuration — this is
+a code deploy only.
+
+1. Deploy the hub and each trust, in any order and at any pace.
+2. When every host is upgraded, set `AES_ACCEPT_LEGACY_CBC=false` and confirm
+   nothing breaks — that proves no peer is still sending CBC.
+3. Delete the shim (`_decrypt_legacy_cbc`, `_accept_legacy_cbc`, the fallback
+   branch in `decrypt`, and the CBC imports) from all four services.
+
+One caveat during the upgrade: the hub marks a task `IN_PROGRESS` when it
+dispatches it, so a task collected by a peer that cannot read it stays
+`IN_PROGRESS`. Reset any such rows to `PENDING` afterwards.
+
+## Rotation
+
+Because every payload names its key, receivers can hold several keys at once:
+add the new key to the keyring, switch the sender to it, then drop the old one.
+No simultaneous change across the hub and every trust.
+
+Note that some ciphertext is stored, not just in flight — queued `TrustTask`
+payloads and `User.encrypted_password` — so re-encrypt or drain those before
+removing a key they were encrypted under.

flip-api/src/flip_api/cohort_services/submit_cohort_query.py

@@ -30,7 +30,7 @@
     TrustDetails,
 )
 from flip_api.domain.schemas.status import ProjectStatus, TaskType
-from flip_api.utils.encryption import encrypt
+from flip_api.utils.encryption import encrypt, kid_for_trust
 from flip_api.utils.logger import logger
 from flip_api.utils.project_manager import has_project_status
 
@@ -207,13 +207,17 @@ def submit_cohort_query(
         result: list[TrustDetails] = []
         queried_trust_ids: list[UUID] = []
 
-        # Encrypt project_id before sending to trusts
-        encrypted_project_id = encrypt(str(cohort_query.project_id))
-        logger.debug("Checking if project_id is encrypted: %s", encrypted_project_id)
-
-        # Queue a task for each trust (instead of direct HTTP calls)
+        # Queue a task for each trust (instead of direct HTTP calls). The
+        # project_id is encrypted per trust under that trust's key-id — a single
+        # ciphertext reused across trusts could not be bound to a per-trust key
+        # (see docs/aes-payload-keys.md). Trusts without their own key
+        # fall back to the shared kid.
         for trust in trusts:
             try:
+                encrypted_project_id = encrypt(
+                    str(cohort_query.project_id),
+                    kid=kid_for_trust(trust_id=str(trust.id), trust_code=getattr(trust, "code", None)),
+                )
                 task_payload = SubmitCohortQueryBody(
                     query_name=query_row.name,
                     query=query_row.query,

flip-api/src/flip_api/domain/interfaces/trust.py

@@ -69,6 +69,15 @@ class ICreatedTrust(BaseModel):
     exactly once. The hub only stores the SHA-256 of the api key; the internal
     service key is not persisted (only used by trust-internal services).
 
+    `trust_aes_key` / `trust_aes_kid` are the trust's payload-encryption key and its
+    key-id. Returned exactly once and stored nowhere hub-side — unlike the api key a
+    symmetric key cannot be reduced to a hash, so an admin who does not record it here
+    has to re-register the trust to get another. Carried for the same reason as the
+    other credentials: this response is the source for the Kit-credentials block of
+    ``trust/.env.<CODE>.<env>``, and the key is one of those credentials. It stays
+    inert until an operator also adds it to the hub's ``AES_TRUST_KEYS`` map
+    (``docs/aes-payload-keys.md``); until then payloads use the shared key.
+
     `fl_kit_slot` is the pre-provisioned FL participant identity assigned to this
     trust from the shared pool. The operator's containers mount the matching
     ``workspace/net-N/services/<fl_kit_slot>/`` provisioned kit dirs; this is the
@@ -82,6 +91,8 @@ class ICreatedTrust(BaseModel):
     created_at: datetime | None = None
     trust_api_key: str
     trust_internal_service_key: str
+    trust_aes_key: str
+    trust_aes_kid: str
     fl_kit_slot: str
     fl_kit_slot_number: int
 

flip-api/src/flip_api/fl_services/services/fl_service.py

@@ -428,6 +428,10 @@ def start_training(
     from flip_api.fl_services.services import fl_scheduler_service
 
     required_info = fl_scheduler_service.get_required_training_details(model_id, session)
+    # Deliberately the shared key, not a per-trust one: this single ciphertext is handed
+    # to the FL server, which fans it out to every participating client. Encrypting it
+    # under one trust's key would make it undecryptable for all the others. Narrowing
+    # this needs the FL server to carry a per-client payload, not a different kid here.
     encrypted_project_id = encrypt(required_info.project_id)
 
     training_details = IStartTrainingBody(

flip-api/src/flip_api/main.py

@@ -18,13 +18,15 @@
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse
 from slowapi.errors import RateLimitExceeded
+from sqlmodel import Session
 
 from flip_api.cohort_services import (
     get_cohort_query_results,
     save_cohort_query,
     submit_cohort_query,
 )
 from flip_api.config import get_settings
+from flip_api.db.database import get_engine
 from flip_api.file_services import (
     delete_file,
     download_file,
@@ -88,6 +90,7 @@
     trusts_health_check,
     update_trust_status,
 )
+from flip_api.trusts_services.services.trust_key_config import log_trust_key_config
 from flip_api.user_services import (
     access_request,
     delete_user,
@@ -101,6 +104,7 @@
     update_user,
 )
 from flip_api.utils.cognito_helpers import get_cors_allowed_origins
+from flip_api.utils.logger import logger
 from flip_api.utils.rate_limiter import limiter
 from flip_api.utils.security_headers import SecurityHeadersMiddleware
 
@@ -120,6 +124,14 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
         app (FastAPI): The FastAPI application instance being started.
     """
     _cors_allowed_origins.extend(get_cors_allowed_origins())
+    # Report per-trust payload-key coverage, and fail fast on a malformed
+    # AES_TRUST_KEYS rather than midway through the first request that encrypts.
+    # A DB outage must not stop the app booting — the audit is diagnostics.
+    try:
+        with Session(get_engine()) as session:
+            log_trust_key_config(session)
+    except Exception:
+        logger.exception("Could not audit the per-trust payload-key configuration at startup")
     start_scheduler()
     print("Starting up the app...")
     yield

flip-api/src/flip_api/private_services/trust_tasks.py

@@ -33,7 +33,7 @@
 from flip_api.domain.schemas.private import TaskResultInput, TrustHeartbeatInput, TrustTaskResponse
 from flip_api.domain.schemas.status import TaskStatus, TaskType
 from flip_api.private_services.imaging_notifications import handle_imaging_task_completed
-from flip_api.utils.encryption import encrypt
+from flip_api.utils.encryption import encrypt, kid_for_trust
 from flip_api.utils.logger import logger
 from flip_api.utils.rate_limiter import limiter
 
@@ -89,6 +89,10 @@ def _get_pending_tasks(trust: Trust, db: Session) -> dict[str, object]:
             return {**_trust_identity(trust), "tasks": []}
 
         now = datetime.now(timezone.utc)
+        # Every task in this batch goes to one trust, so the whole batch is encrypted
+        # under that trust's key when it has one. Resolved once rather than per task:
+        # the lookup is keyring-only, and the answer cannot change within a batch.
+        kid = kid_for_trust(trust_id=str(trust.id), trust_code=trust.code)
         response: list[TrustTaskResponse] = []
         for task in tasks:
             task.status = TaskStatus.IN_PROGRESS
@@ -97,7 +101,7 @@ def _get_pending_tasks(trust: Trust, db: Session) -> dict[str, object]:
                 TrustTaskResponse(
                     id=task.id,
                     task_type=task.task_type,
-                    payload=encrypt(task.payload),
+                    payload=encrypt(task.payload, kid=kid),
                     created_at=task.created_at,
                 )
             )

flip-api/src/flip_api/scripts/generate_trust_key.py

@@ -17,6 +17,7 @@
 are never added by hand; ``register_trust`` is the sole writer of the registry.
 """
 
+import base64
 import hashlib
 import secrets
 
@@ -30,3 +31,17 @@ def generate_trust_key() -> tuple[str, str]:
     key = secrets.token_urlsafe(32)
     key_hash = hashlib.sha256(key.encode()).hexdigest()
     return key, key_hash
+
+
+def generate_aes_key() -> str:
+    """Generate a fresh 256-bit AES key, base64-encoded.
+
+    Used by ``register_trust`` to mint a per-trust payload-encryption key.
+    Unlike the API key, this is a symmetric key the hub must also hold to
+    encrypt to / decrypt from the trust, so — like ``AES_KEY_BASE64`` — only
+    the plaintext is meaningful (there is no hash form).
+
+    Returns:
+        str: Base64-encoded 32-byte (AES-256) key.
+    """
+    return base64.b64encode(secrets.token_bytes(32)).decode()

flip-api/src/flip_api/scripts/register_trust.py

@@ -175,6 +175,12 @@ def register_one_trust(
             "trust_name": kit.trust.name,
             "trust_api_key": kit.trust_api_key,
             "trust_internal_service_key": kit.trust_internal_service_key,
+            # Per-trust payload-encryption key + kid. Belongs in the
+            # trust's kit as TRUST_AES_KEY_BASE64 / TRUST_AES_KID; inert until the
+            # hub also holds it (docs/aes-payload-keys.md), so until then payloads
+            # keep using the shared key.
+            "trust_aes_key": kit.trust_aes_key,
+            "trust_aes_kid": kit.trust_aes_kid,
             "fl_kit_slot": kit.fl_kit_slot.slot_name,
             "fl_kit_slot_number": kit.fl_kit_slot.slot_number,
             "hub_shared": _hub_shared_from_env(),

flip-api/src/flip_api/trusts_services/admin_create_trust.py

@@ -103,6 +103,8 @@ def admin_create_trust(
         created_at=registered.trust.created_at,
         trust_api_key=registered.trust_api_key,
         trust_internal_service_key=registered.trust_internal_service_key,
+        trust_aes_key=registered.trust_aes_key,
+        trust_aes_kid=registered.trust_aes_kid,
         fl_kit_slot=registered.fl_kit_slot.slot_name,
         fl_kit_slot_number=registered.fl_kit_slot.slot_number,
     )

flip-api/src/flip_api/trusts_services/services/register_trust.py

@@ -36,7 +36,7 @@
 from flip_api.db.models.main_models import FLKitSlot, Trust
 from flip_api.db.seed.fl_kit_slots import insert_missing_slots, resolve_fl_kit_slot_names
 from flip_api.domain.schemas.actions import TrustAuditAction
-from flip_api.scripts.generate_trust_key import generate_trust_key
+from flip_api.scripts.generate_trust_key import generate_aes_key, generate_trust_key
 from flip_api.trusts_services.utils.audit_helper import audit_trust_action
 from flip_api.utils.logger import logger
 
@@ -68,12 +68,20 @@ class RegisteredTrust:
     Plaintext ``trust_api_key`` and ``trust_internal_service_key`` are returned
     exactly once: the hub stores only the api-key's SHA-256 hash, and the
     internal-service key is never persisted hub-side.
+
+    ``trust_aes_key`` / ``trust_aes_kid`` are the per-trust payload-encryption key
+    and its key-id. This is a symmetric key the hub must also hold to
+    encrypt to / decrypt from this trust, so — unlike the api key — it cannot be
+    reduced to a hash. Until it is provisioned on both sides
+    (``docs/aes-payload-keys.md``) payloads keep using the shared key.
     """
 
     trust: Trust
     fl_kit_slot: FLKitSlot
     trust_api_key: str
     trust_internal_service_key: str
+    trust_aes_key: str
+    trust_aes_kid: str
 
 
 def _claim_free_slot(session: Session) -> FLKitSlot | None:
@@ -216,4 +224,6 @@ def register_trust(
         fl_kit_slot=slot,
         trust_api_key=api_key,
         trust_internal_service_key=internal_key,
+        trust_aes_key=generate_aes_key(),
+        trust_aes_kid=f"trust-{trust.id}",
     )