Skip to content
Open
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
bfb6a2a
feat(deploy/AWS): add the PROD=lza environment plumbing for the LZA a…
atriaybagur Jul 9, 2026
e317c47
feat(deploy/AWS): conditional network layer for the LZA platform-mana…
atriaybagur Jul 9, 2026
65f052b
feat(deploy/AWS): gate the SG-drift stack and public FL NLB off on LZ…
atriaybagur Jul 9, 2026
84a3b48
feat(deploy/AWS): support a DNS-less first bring-up on the default Cl…
atriaybagur Jul 9, 2026
c2d2a97
docs(deploy/AWS): document the PROD=lza deployment path (#749)
atriaybagur Jul 9, 2026
8dcdf5a
feat(deploy/AWS): route the Flower register-keys one-shot through the…
atriaybagur Aug 17, 2026
71e13ab
docs(deploy/AWS): refresh the LZA state notes — multi-AZ landed, FL i…
atriaybagur Aug 17, 2026
db86cfe
refactor(deploy/AWS): short 'lza-prod' profile alias for the LZA guar…
atriaybagur Aug 17, 2026
5a9343f
docs(deploy/AWS): correct the multi-AZ state — data-b and tgw-b subne…
atriaybagur Aug 17, 2026
512e19e
fix(deploy/AWS): gate the endpoint-SG trust-egress rule off on LZA (#…
atriaybagur Aug 17, 2026
a492b9b
feat(deploy/AWS): overridable log-bucket names for cross-account name…
atriaybagur Aug 17, 2026
fbf6c27
feat(deploy/AWS): port the LZA FL/web ingress from the e2e harness (#…
atriaybagur Aug 19, 2026
d4cc046
feat(deploy/AWS): gate the workload CloudFront off on LZA (#749 WP3)
atriaybagur Aug 20, 2026
1fa5739
fix(flip-api): never re-resolve the venv at boot (UV_NO_SYNC)
atriaybagur Aug 20, 2026
4f914e5
feat(deploy/AWS): skip the CloudFront invalidation in deploy-ui on LZA
atriaybagur Aug 20, 2026
52bfb5c
fix(flip-api): put src/ on PYTHONPATH now the boot-time sync is gone
atriaybagur Aug 20, 2026
f05c8e6
fix(deploy/AWS): classic hosted UI on LZA -- Managed Login blocks Pri…
atriaybagur Aug 20, 2026
6aa2e5e
fix(deploy/AWS): pin the LZA user pool to the LITE tier
atriaybagur Aug 20, 2026
0f41212
fix(deploy/AWS): no hosted-UI domain on LZA -- it latches the Private…
atriaybagur Aug 20, 2026
16c0edc
fix(deploy/AWS): admit the VPC CIDR on the LZA ALB web rule
atriaybagur Aug 20, 2026
f647cd4
fix(deploy/AWS): admit the internal FL NLB onto the fl-server task SG
atriaybagur Aug 20, 2026
ac31270
Merge develop into 749-lza-terraform-env (pick up the MFA first-login…
atriaybagur Aug 20, 2026
2dcbaa1
fix(lza): pin fl-server-net-1 to the internal FL NLB static IPs for f…
atriaybagur Aug 20, 2026
fe6d935
Revert "fix(lza): pin fl-server-net-1 to the internal FL NLB static I…
atriaybagur Aug 20, 2026
646a3f8
fix(lza): resolve the admin kit's bare fl-server-net-1 via a private …
atriaybagur Aug 20, 2026
07163fa
fix(lza): codify the accelerator-managed ALB access logging
atriaybagur Aug 20, 2026
f53e81c
fix(deploy/AWS): access_logs 'off' is null, not {} -- empty map broke…
atriaybagur Aug 21, 2026
0d6af84
docs(deploy/AWS): document the two-phase LZA edge wiring; refresh WP1…
atriaybagur Aug 21, 2026
7cd6b5b
fix(fl-api-flower): drive the flwr CLI from the baked venv, not uvx
atriaybagur Aug 21, 2026
eb12a65
feat(deploy/AWS): disable Flower runtime dependency installs on LZA
atriaybagur Aug 21, 2026
9946924
docs(deploy/AWS): frame the two deployment modes; refresh LZA state n…
atriaybagur Aug 24, 2026
1eae093
docs(deploy/AWS): confirm the multi-AZ state against the live account
atriaybagur Aug 24, 2026
a74c091
docs: cover both Central Hub deployment modes in the published guide
atriaybagur Aug 24, 2026
ef6456e
feat(deploy/AWS): PROD=lza-stag — staging on a platform-managed estate
atriaybagur Aug 26, 2026
4c81a70
Merge branch 'develop' into 749-lza-terraform-env
atriaybagur Aug 28, 2026
84a577a
fix(deploy/AWS): teach register-trusts.sh and add_fl_kits.sh the LZA …
atriaybagur Aug 28, 2026
ac1951c
fix(fl-services): load .env.lza-prod / .env.lza-stag for the kit-uplo…
atriaybagur Aug 28, 2026
d5d74b6
fix(make): name LZA trust kits .lza-prod / .lza-stag from the root ki…
atriaybagur Aug 28, 2026
e6cda87
docs(deploy/AWS): list the raw TF_VAR exports an LZA env file carries
atriaybagur Aug 28, 2026
852590d
Merge branch 'develop' into 749-lza-terraform-env (pick up the checko…
atriaybagur Aug 28, 2026
e1ee31e
fix(deploy/AWS): satisfy the checkov lint on fl_ingress_lza.tf
atriaybagur Aug 28, 2026
09c3782
docs: how a real domain attaches to the LZA edge (alias + us-east-1 ACM)
atriaybagur Sep 1, 2026
6490a58
docs: the fresh-account Secrets Manager/KMS-alias bootstrap trap
atriaybagur Sep 1, 2026
80b601c
docs: make the LZA out-of-band prerequisites reusable per account
atriaybagur Sep 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,7 @@ flip-fl-api/transfer/*/*
.env.development
.env.stag
.env.production
.env.lza-prod
# Per-trust operator kits — one file per trust, named by trust CODE and
# environment (e.g. trust/.env.KCH.development, trust/.env.<CODE>.production).
# Minted by the hub via register-trust / POST /admin/trusts; contain plaintext
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ After changes, evaluate if docs need updating:
- `FL_PROVISIONED_DIR` — path to the in-tree provisioned FL artifacts, derived per-backend by `deploy/fl_backend.mk` from `FL_BACKEND`: `fl-services/nvflare/provision/workspace-dev` (nvflare startup kits) or `fl-services/flower/provision/creds` (flower per-net TLS certs + SuperNode keys). Both gitignored. Read only by the dev compose overlays for the cert/workspace volume mounts; override at the CLI for a one-off (`make up FL_PROVISIONED_DIR=...`). FL Makefiles are **per-backend** — each `fl-services/<backend>/Makefile` owns that backend's `build`/`provision`/`up`/`down`/`submit` (flower also `up-secure`); the root Makefile forwards only `build-fl` by `FL_BACKEND`. Each backend's `fl-services/<backend>/Makefile` also owns its network provisioning (NVFLARE adds `provision`/`provision-2-nets`/`provision-stag`/`provision-prod`/`upload-kits-to-s3`; the project YAMLs, `scripts/`, and gitignored `workspace-{dev,stag,prod}/` output live under `provision/`). Provision with `make -C fl-services/nvflare provision-2-nets` (nvflare) or `make -C fl-services/flower provision NET_NUMBER=<N>` (flower). To run a backend standalone + submit without the full stack: `make -C fl-services/<backend> up` (or `up-secure`) then `make -C fl-services/<backend> submit APP=<job>`.
- `FL_APP_BASE_DIR` — Local directory holding the base FL application templates (the repo's `fl-apps/` tree), baked into the flip-api image and bind-mounted in dev. flip-api walks `<FL_APP_BASE_DIR>/<backend>/<job_type>/` to bundle an application (uploading those files into `FL_APP_DESTINATION_BUCKET/<model_id>`) and reads each backend's manifest from `<FL_APP_BASE_DIR>/<backend>/required_files.json`. Default `/app/fl-apps`; override to mount operator-provided templates. Replaces the removed `FL_APP_BASE_BUCKET` S3 dependency (FLIP#724): base templates are no longer published to S3 (the `fl-apps-push-s3-*` sync workflows are gone), so a template hotfix now ships by rebuilding + redeploying the flip-api image rather than syncing S3. `fl-apps/` is baked into the image via a BuildKit named build context (`fl_apps=../fl-apps`) since it sits outside flip-api's build context. For the Flower backend, the template pyprojects also steer Flower's **per-run dependency install** (`uv sync` on every app launch; SuperNodes opt in via `--allow-runtime-dependency-installation` in the composes): `[tool.uv.sources]` pins `flip-utils` to the source kept at `/opt/flip-utils` inside the FL images (never PyPI — FLIP#767; a flip-utils change ships by rebuilding the FL images, `make build-fl FL_BACKEND=flower`) and torch/torchvision to the cu128 index (PyPI's default cu130 wheels need driver >=580).
- `FL_KIT_SLOT_NAMES` — JSON list (e.g. `["Trust_1", "Trust_2"]`) of FL kit-slot names for the hub's `fl_kit_slot` pool that `register_trust` claims from; each name must match a provisioned participant kit (in-tree workspace for dev, `s3://<AICENTRE_BUCKET_NAME>/fl-flare-participant-kits/<FLARE_KIT_DATE>/net-<N>/services/<slot>/` for stag/prod; slot names are global across nets — every net carries a kit per name). The pool is seeded at flip-api boot and **reconciled on demand** when a registration finds it exhausted (`resolve_fl_kit_slot_names`, additive — never deletes or re-assigns rows); only then does `NoFreeKitSlotError` surface. Single source per env: dev = this env var (a `DevSettings`-only field; restart to change, settings load once); stag/prod = the `/flip/fl_kit_slot_names` SSM parameter (Terraform-rendered from this var — the list is plain config, not a secret; deliberately **no env fallback**, so a broken/missing parameter means the pool can't grow, loudly, never masked by stale task-def env). Growing the pool is an env-file edit + `make -C deploy/providers/AWS apply-fl-kit-slots` (targeted plan/apply of just the parameter, plain-text diff) — **no restart, no task-definition change**. One-command workflow: `make -C deploy/providers/AWS add-fl-kits N=<n> PROD=stag|true` (N = "ensure N more live slots": activate spares toward N first, mint only the shortfall on every net → additive S3 upload → env edit → parameter apply); full runbook in `fl-services/nvflare/README.md` ("Onboarding a new client onto an existing network"). NVFLARE-only dynamics — Flower's SuperNode key labelling reads the list at net startup.
- `PROD` — `true` (production), `stag` (staging), unset (development)
- `PROD` — `true` (production), `stag` (staging), `lza` (LZA FLIPProduction account, FLIP#749 — meaningful for `deploy/providers/AWS` targets only; selects the root `.env.lza-prod` and the platform-managed-network Terraform path, see `deploy/providers/AWS/README.md` "Deploying to the LZA account"), unset (development)
- `AES_KEY_BASE64` — encryption key for trust communication
- A remote trust operator only needs their kit file (`trust/.env.<KIT>`) — no hub `.env.<env>` needed on trust hosts.
See `trust/README.md` for the standalone-operator quick-start.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ After changes, evaluate if docs need updating:
- `FL_PROVISIONED_DIR` — path to the in-tree provisioned FL artifacts, derived per-backend by `deploy/fl_backend.mk` from `FL_BACKEND`: `fl-services/nvflare/provision/workspace-dev` (nvflare startup kits) or `fl-services/flower/provision/creds` (flower per-net TLS certs + SuperNode keys). Both gitignored. Read only by the dev compose overlays for the cert/workspace volume mounts; override at the CLI for a one-off (`make up FL_PROVISIONED_DIR=...`). FL Makefiles are **per-backend** — each `fl-services/<backend>/Makefile` owns that backend's `build`/`provision`/`up`/`down`/`submit` (flower also `up-secure`); the root Makefile forwards only `build-fl` by `FL_BACKEND`. Each backend's `fl-services/<backend>/Makefile` also owns its network provisioning (NVFLARE adds `provision`/`provision-2-nets`/`provision-stag`/`provision-prod`/`upload-kits-to-s3`; the project YAMLs, `scripts/`, and gitignored `workspace-{dev,stag,prod}/` output live under `provision/`). Provision with `make -C fl-services/nvflare provision-2-nets` (nvflare) or `make -C fl-services/flower provision NET_NUMBER=<N>` (flower). To run a backend standalone + submit without the full stack: `make -C fl-services/<backend> up` (or `up-secure`) then `make -C fl-services/<backend> submit APP=<job>`.
- `FL_APP_BASE_DIR` — Local directory holding the base FL application templates (the repo's `fl-apps/` tree), baked into the flip-api image and bind-mounted in dev. flip-api walks `<FL_APP_BASE_DIR>/<backend>/<job_type>/` to bundle an application (uploading those files into `FL_APP_DESTINATION_BUCKET/<model_id>`) and reads each backend's manifest from `<FL_APP_BASE_DIR>/<backend>/required_files.json`. Default `/app/fl-apps`; override to mount operator-provided templates. Replaces the removed `FL_APP_BASE_BUCKET` S3 dependency (FLIP#724): base templates are no longer published to S3 (the `fl-apps-push-s3-*` sync workflows are gone), so a template hotfix now ships by rebuilding + redeploying the flip-api image rather than syncing S3. `fl-apps/` is baked into the image via a BuildKit named build context (`fl_apps=../fl-apps`) since it sits outside flip-api's build context. For the Flower backend, the template pyprojects also steer Flower's **per-run dependency install** (`uv sync` on every app launch; SuperNodes opt in via `--allow-runtime-dependency-installation` in the composes): `[tool.uv.sources]` pins `flip-utils` to the source kept at `/opt/flip-utils` inside the FL images (never PyPI — FLIP#767; a flip-utils change ships by rebuilding the FL images, `make build-fl FL_BACKEND=flower`) and torch/torchvision to the cu128 index (PyPI's default cu130 wheels need driver >=580).
- `FL_KIT_SLOT_NAMES` — JSON list (e.g. `["Trust_1", "Trust_2"]`) of FL kit-slot names for the hub's `fl_kit_slot` pool that `register_trust` claims from; each name must match a provisioned participant kit (in-tree workspace for dev, `s3://<AICENTRE_BUCKET_NAME>/fl-flare-participant-kits/<FLARE_KIT_DATE>/net-<N>/services/<slot>/` for stag/prod; slot names are global across nets — every net carries a kit per name). The pool is seeded at flip-api boot and **reconciled on demand** when a registration finds it exhausted (`resolve_fl_kit_slot_names`, additive — never deletes or re-assigns rows); only then does `NoFreeKitSlotError` surface. Single source per env: dev = this env var (a `DevSettings`-only field; restart to change, settings load once); stag/prod = the `/flip/fl_kit_slot_names` SSM parameter (Terraform-rendered from this var — the list is plain config, not a secret; deliberately **no env fallback**, so a broken/missing parameter means the pool can't grow, loudly, never masked by stale task-def env). Growing the pool is an env-file edit + `make -C deploy/providers/AWS apply-fl-kit-slots` (targeted plan/apply of just the parameter, plain-text diff) — **no restart, no task-definition change**. One-command workflow: `make -C deploy/providers/AWS add-fl-kits N=<n> PROD=stag|true` (N = "ensure N more live slots": activate spares toward N first, mint only the shortfall on every net → additive S3 upload → env edit → parameter apply); full runbook in `fl-services/nvflare/README.md` ("Onboarding a new client onto an existing network"). NVFLARE-only dynamics — Flower's SuperNode key labelling reads the list at net startup.
- `PROD` — `true` (production), `stag` (staging), unset (development)
- `PROD` — `true` (production), `stag` (staging), `lza` (LZA FLIPProduction account, FLIP#749 — meaningful for `deploy/providers/AWS` targets only; selects the root `.env.lza-prod` and the platform-managed-network Terraform path, see `deploy/providers/AWS/README.md` "Deploying to the LZA account"), unset (development)
- `AES_KEY_BASE64` — encryption key for trust communication
- A remote trust operator only needs their kit file (`trust/.env.<KIT>`) — no hub `.env.<env>` needed on trust hosts.
See `trust/README.md` for the standalone-operator quick-start.
Expand Down
6 changes: 5 additions & 1 deletion deploy/providers/AWS/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
| File | Resources |
|------|-----------|
| `main.tf` | Provider config, VPC, subnets, IGW, NAT, route tables, RDS instance, Secrets Manager, SES |
| `network_lza.tf` | LZA platform-managed network (FLIP#749): VPC/subnet data lookups + the `local.vpc_id` / `local.app_subnet_ids` / `local.data_subnet_ids` locals both paths consume |
| `fl_ingress_lza.tf` | LZA-only ingress (FLIP#749 WP3, ported from the #829 e2e harness): internal FL NLB with static per-subnet IPs + its TG/SG, the ALB ingress rule for the networking-account relay path, and the `/flip/networking/*` edge-handoff SSM params |
| `services.tf` | S3 buckets, Cognito |
| `rds_proxy.tf` | RDS Proxy + IAM DB auth (proxy, IAM role/policy, SG, `rds-db:connect`) — see FLIP#556 |
| `ecs.tf` | ECS cluster, capacity providers, ECS CloudWatch log groups (ALB / NLB / target groups / listener rules live in `main.tf`) |
Expand All @@ -27,13 +29,15 @@
| ------- | ------------- | --------- |
| `stag` | Staging | `flipstag` |
| `prod` | Production | `flipprod` |
| `FlipDeveloperAccess-080369786334` | Developer access | — |
| `lza-prod` | LZA FLIPProduction (`PROD=lza`, FLIP#749; `FLIPAdminAccess` permission set) | `893493035022` |
| `dev` | Development (the `dev/` root: Cognito + SES; `FlipDeveloperAccess` permission set) | `flipdev` |

## Key Deploy Commands

```bash
make full-deploy PROD=stag # Full staging deploy
make full-deploy PROD=true # Full prod deploy
make init/plan/apply PROD=lza # LZA FLIPProduction (env-gated; full-deploy chains untested there — see README "Deploying to the LZA account")
make full-deploy-hybrid PROD=<stag|true> [LOCAL_TRUST_IP=<ip>] # Hybrid with on-prem trust
make full-deploy-hub-only PROD=<stag|true> # Hub only, NO cloud Trust EC2 (all trusts on-prem, e.g. GPU hosts) — see README "Hub-only Deployment"
make init/plan/apply # Terraform workflow
Expand Down
6 changes: 5 additions & 1 deletion deploy/providers/AWS/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
| File | Resources |
|------|-----------|
| `main.tf` | Provider config, VPC, subnets, IGW, NAT, route tables, RDS instance, Secrets Manager, SES |
| `network_lza.tf` | LZA platform-managed network (FLIP#749): VPC/subnet data lookups + the `local.vpc_id` / `local.app_subnet_ids` / `local.data_subnet_ids` locals both paths consume |
| `fl_ingress_lza.tf` | LZA-only ingress (FLIP#749 WP3, ported from the #829 e2e harness): internal FL NLB with static per-subnet IPs + its TG/SG, the ALB ingress rule for the networking-account relay path, and the `/flip/networking/*` edge-handoff SSM params |
| `services.tf` | S3 buckets, Cognito |
| `rds_proxy.tf` | RDS Proxy + IAM DB auth (proxy, IAM role/policy, SG, `rds-db:connect`) — see FLIP#556 |
| `ecs.tf` | ECS cluster, capacity providers, ECS CloudWatch log groups (ALB / NLB / target groups / listener rules live in `main.tf`) |
Expand All @@ -27,13 +29,15 @@
| ------- | ------------- | --------- |
| `stag` | Staging | `flipstag` |
| `prod` | Production | `flipprod` |
| `FlipDeveloperAccess-080369786334` | Developer access | — |
| `lza-prod` | LZA FLIPProduction (`PROD=lza`, FLIP#749; `FLIPAdminAccess` permission set) | `893493035022` |
| `dev` | Development (the `dev/` root: Cognito + SES; `FlipDeveloperAccess` permission set) | `flipdev` |

## Key Deploy Commands

```bash
make full-deploy PROD=stag # Full staging deploy
make full-deploy PROD=true # Full prod deploy
make init/plan/apply PROD=lza # LZA FLIPProduction (env-gated; full-deploy chains untested there — see README "Deploying to the LZA account")
make full-deploy-hybrid PROD=<stag|true> [LOCAL_TRUST_IP=<ip>] # Hybrid with on-prem trust
make full-deploy-hub-only PROD=<stag|true> # Hub only, NO cloud Trust EC2 (all trusts on-prem, e.g. GPU hosts) — see README "Hub-only Deployment"
make init/plan/apply # Terraform workflow
Expand Down
Loading
Loading