Skip to content

feat(deploy/AWS): env-gated PROD=lza deployment path for the LZA FLIPProduction account (#749 WP1-WP3) - #979

Open
atriaybagur wants to merge 41 commits into
developfrom
749-lza-terraform-env
Open

feat(deploy/AWS): env-gated PROD=lza deployment path for the LZA FLIPProduction account (#749 WP1-WP3)#979
atriaybagur wants to merge 41 commits into
developfrom
749-lza-terraform-env

Conversation

@atriaybagur

@atriaybagur atriaybagur commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

Adds an env-gated PROD=lza deployment path targeting an LZA-provisioned workload account (the AI Centre's FLIPProduction, reached through the lza-prod profile alias) — WP1 of the LZA migration, plus the WP3 ingress port and the fixes found bringing the stack live. The legacy PROD=true/PROD=stag paths are byte-identical no-ops (proof below): LZA support is a third option alongside them, not a replacement, per the #749 coexistence guarantee. The docs now present this explicitly as two permanently supported deployment modes — self-contained single-account (the open-source shape) and platform-managed/LZA — rather than a migration with an end date.

What PROD=lza selects (full docs in deploy/providers/AWS/README.md → "Deploying onto an LZA estate"):

  • Env plumbing: root .env.lza-prod (gitignored), AWS_PROFILE guard defaulting to the short lza-prod alias (the account's FLIPAdminAccess permission set, same convention as prod/stag), separate trust/.env.<CODE>.lza-prod kit namespace.
  • Two orthogonal axes: TF_VAR_environment=prod (LZA is a production estate — all prod hardening stays on) + the new TF_VAR_lza_managed_network=true (platform-managed network).
  • Conditional network layer (network_lza.tf): VPC-layer creation is SCP-denied in the LZA account, so the VPC module empties via its create flag and consumers switch to Name-tag data lookups of the accelerator-provisioned VPC/subnets — RDS into the isolated data subnets, everything else into the TGW-routed app subnets. Subnet lookups match all -app-*/-data-* hits, so platform-added AZs appear on the next plan without code changes.
  • LZA FL/web ingress (fl_ingress_lza.tf, ported from the [Feature]: e2e-lza — LZA ingress end-to-end test stack (workload-account half) #829/[Feature]: e2e-lza — LZA ingress end-to-end test stack (workload-account half) #830 e2e harness and now serving the real stack): internal FL NLB with deterministic static per-subnet IPs (the stable targets the networking-account edge NLB registers once — no sync machinery), backend-keyed Fargate TG, SG admissions for the edge relay + in-VPC callers, and /flip/networking/* SSM handoff params for the networking account. fl-server-net-1 now always attaches a load balancer — the legacy TG or the LZA one.
  • Workload CloudFront gated off: the GRCLOUDFRONTVPCORIGIN SCP denies VPC origins by design (they'd bypass the TGW + central firewall). The networking account's edge distribution (aicentre-lza-iac) is the front door — it serves the UI bucket via cross-account OAC and relays /api/* through the firewall to the internal ALB. deploy-ui skips the CloudFront invalidation on LZA (immutable hashed bundles + no-cache index.html make it unnecessary, and no cross-account invalidation permission is wanted).
  • Cognito on the egress-less account: Cognito's PrivateLink gate refuses interface-endpoint API access to Managed-Login pools, and the interface endpoint is the account's only route to cognito-idp — so the LZA pool runs the classic hosted UI (v1), pins the LITE tier, and creates no hosted-UI domain (each of the three latches the block independently; found empirically 2026-08-20). FLIP never uses the hosted UI (SDK sign-in), so all three are cosmetic no-ops functionally. Legacy pools unchanged.
  • Flower CLI offline-safe (fl-api-flower): every flwr invocation (run/list/stop/federation list) went through uvx, which resolves its tool environment from PyPI at call time — impossible on the egress-less account (submissions errored the model; run listing, status and the [Bug]: a Flower run that fails after submission is invisible — model sits at INITIATED with no error #1001 reconcile went dark). Now invokes the service venv's own flwr binary (already a direct dependency, on PATH in the image); no call-time network on any environment.
  • flip-api boot without egress: UV_NO_SYNC makes the build-time venv authoritative — the entrypoint's uv run used to re-resolve the project every boot, fetching the setuptools backend from PyPI, which the egress-less account can't reach (crash-loop) — plus PYTHONPATH=src since the boot-time re-sync was also what installed the package. One mechanism for prod images and dev bind-mounts alike; legacy just stops paying a wasted PyPI fetch per boot.
  • Bare fl-server-net-1 resolution: the NVFLARE admin kit targets the bare name, which legacy resolves via the flip.local DHCP search domain (DHCP options are LZA-gated off, and Fargate's awsvpc mode rejects extraHosts) — so LZA publishes it as a single-label private hosted zone, apex A records on the internal FL NLB's static IPs.
  • Accelerator guardrails codified: ALB access logging to the LogArchive bucket (lza_elb_access_logs_bucket, empty default omits the block) so plans stop proposing to revert the out-of-band enable; overridable log-bucket names for cross-account name collisions.
  • Gated off on LZA: in-account VPC endpoints, DHCP options, the legacy /flip/networking/* TGW coupling, the SG-drift CloudTrail stack (org baseline covers it), and the public internet-facing FL NLB (no IGW + VPC BPA — superseded by the internal NLB + edge path above). State safety everywhere via create flags / count + moved blocks, so legacy state addresses never churn.
  • DNS-less first bring-up (MANAGE_DNS=false): no hosted zone in the account yet, so the edge distribution runs on its default domain with local.ui_origin feeding bucket CORS + Cognito URLs; all of it reverts by flipping the toggle once the zone lands.
  • Docs — two deployment modes, not a migration: deploy/providers/AWS/README.md and deploy/README.md now present the Terraform root as offering two permanently supported modes (self-contained single-account vs platform-managed/LZA) rather than a transition with an end date, and the published Sphinx guide (docs/source/deploy-flip/deploy-central-hub.rst) covers the second mode for the first time — it previously described only the self-contained shape (own VPC, in-account CloudFront, public FL NLB, .env.stag/.env.production), every line of which is false under PROD=lza. Operator detail stays in the AWS README; the published page says the mode exists, what it implies, and where to read on. Two stale caveats in the LZA section were corrected against the live account: multi-AZ has landed (all six app/data/tgw subnets exist, flip-db-subnet-group spans both AZs, the TGW attachment rides both tgw subnets) and FL ingress is wired (fl_ingress_lza.tf) — the section had claimed neither, while contradicting itself on the second.
  • PROD=lza-stag — staging on a platform-managed estate: the LZA mode is a family, not one environment. lza-stag composes TF_VAR_environment=stag with lza_managed_network=true (the axes were already orthogonal, so zero Terraform changes): env file .env.lza-stag, profile guard lza-stag, kit suffix .lza-stag, origin/develop deploys, prod-only RDS hardening off. LZA_VPC_NAME is mandatory there (the variables.tf default names the prod VPC template) — the Makefile refuses to run without it rather than letting the data lookup fail opaquely. One workload account per environment; the staging account itself is a platform-side ask tracked on londonaicentre/lza#33. The helpers that re-derive the environment from PROD on their own — register-trusts.sh, add_fl_kits.sh, the fl-services/<backend> kit-upload Makefiles and the root kit-file targets (new-trust, sync-trust-kit) — now know both LZA values, so an lza-stag registration writes the .lza-stag kit the deploy targets read back (they used to fall through to .stag / .env.development); the LZA pair reuses the legacy kit workspaces on purpose (same FQDN + root CA across the DNS cutover). The account itself is now londonaicentre/lza#49 (staging = the estate's Workloads/Test tier, LZA_VPC_NAME=AWSAccelerator-eu-west-2-test).
  • Rebase reconciliation: ecs_flower.tf (landed on develop after this branch was cut) read module.flip_vpc.private_subnets directly — empty on LZA; now routed through the LZA-aware local.app_subnet_ids (a legacy no-op by construction).

Linked Issues

Part of #749 (WP1 + WP3 — deliberately not a closing keyword: the umbrella stays open through WP6). The #829/#830 FL-ingress e2e stack that proved the two-tier path is now retired — this PR's real stack superseded it.

Checklist

  • Follows the project's coding conventions and style guide
  • Updates documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Type of Change

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • In-line docstrings updated.
  • Documentation updated, tested make -C docs/ docs.

Testing

  • Legacy no-op proof: make -n plan PROD=true and PROD=stag are byte-identical (stdout and stderr) between origin/develop and this branch — the LZA gating changes nothing for the legacy environments. The flip-api and fl-api-flower commits are env-agnostic code changes whose only legacy effect is dropping call-time PyPI fetches.
  • fl-api-flower make local_test green after the uvx→venv-flwr change: ruff + mypy + 113 tests passing.
  • terraform fmt -check -recursive clean; terraform validate passes (same checks as the validate_terraform CI job).
  • Live stag plan from this branch (2026-08-21): make plan PROD=stag against the staging estate. This check caught the branch's one legacy regression — the ALB access_logs conditional passed {} where the module's object type (required bucket, default null) demands null, erroring every legacy plan — fixed in f53e81c. With the fix the plan is 4 to add, 4 to change, 0 to destroy, every line traced to develop-ahead drift (KMS-CMK SG tags, trust-EC2 S3 policy narrowing) or out-of-band ops (ECS Exec enables), none to this branch; the sg-drift moved blocks resolve as pure renames.
  • Live on FLIPProduction (2026-08-19 → 08-21): full make apply PROD=lza from this branch, then end-to-end verification against the running stack — UI + API served through the networking-account edge distribution (caching behaviour verified), SDK sign-in with TOTP MFA confirmed, a local trust registered and brought up against the LZA hub over the edge, and a complete NVFLARE training run passed (project → cohort → image pull → training → RESULTS_UPLOADED → results downloaded), exercising the two-tier FL ingress (edge NLB → internal NLB static IPs → fl-server) and the [Bug]: a Flower run that fails after submission is invisible — model sits at INITIATED with no error #1001 failed-run reconcile sweep live.
  • Flower e2e also passed on the live stack (2026-08-21): after the TF-native backend switch, a complete Flower training run — submission via the venv flwr CLI (offline), ServerApp in the baked image env (FLWR_DISABLE_RUNTIME_DEPENDENCY_INSTALLATION), two SuperNodes training through the edge with FLIP local DP, aggregation, RESULTS_UPLOADED, and results downloaded. One known limitation exercised en route: single-trust Flower runs hang on flwr's default node minimums — the dedicated fix is fix(ui,api,fl-apps,flower,trust): stop guessing job types; require config.json for Flower; fix single-trust hang; make the OMOP data dir an argument #991 (which carries fix(flower): derive FedAvg node thresholds from the participating-trust count #992 on its branch; this run used two SuperNodes, and nothing feat(deploy/AWS): env-gated PROD=lza deployment path for the LZA FLIPProduction account (#749 WP1-WP3) #979-side is needed once fix(ui,api,fl-apps,flower,trust): stop guessing job types; require config.json for Flower; fix single-trust hang; make the OMOP data dir an argument #991 merges).
  • PROD-value plumbing: deploy/providers/AWS/scripts/tests/test_add_fl_kits.sh (18 cases, incl. lza-stag → stag env, lza → prod env, unknown value refused) and scripts/tests/test_new_trust.py (21, incl. the two LZA suffixes + their PROD= hints) green; make -n dry-runs of every PROD-keyed target (register-trusts, add-fl-kits, upload-kits-to-s3, upload-creds-to-s3, new-trust, sync-trust-kit) against a stub .env.lza-stag resolve the LZA env file and kit suffix, while PROD=stag|true|unset print exactly what they did before; legacy make -n plan PROD=stag re-proven byte-identical (stdout + stderr) against origin/develop.
  • Checkov lint (ci: checkov security lint in validate_terraform (#1052 + #1058 triage) #1057, merged from develop 2026-08-28): run locally over the merged tree with the CI-pinned checkov==3.3.14 via scripts/checkov_lint.sh — the LZA files were scanned for the first time; fl_ingress_lza.tf gained the ~> 10.0 module version pin (already resolved at 10.5.0, no apply change) and the same CKV2_AWS_34 skips develop applied to the legacy /flip/networking/* handoff params (non-secret, read cross-account). 0 failures; legacy make -n plan PROD=stag re-proven byte-identical against the new develop tip.

Additional Notes

  • Root CLAUDE.md/AGENTS.md and deploy/providers/AWS/CLAUDE.md/AGENTS.md mirrors regenerated in the same commits.
  • No AWS account IDs in tracked files: the LZA workload account is identified by its lza-prod profile alias, the same way prod/stag are.
  • The state-migration moved blocks are annotated as removable once every live state file has crossed the rename.
  • Known follow-up: on LZA, make deploy-centralhub's tag-manifest guard is skipped for the ECR pull-through registry, so it currently repoints the FL services at the hub tag too — needs an FL-service exclusion before routine use there.

@atriaybagur
feat(deploy/AWS): add the PROD=lza environment plumbing for the LZA a…
bfb6a2a 
…ccount (#749)

PROD=lza selects the gitignored root .env.lza-prod, guards on the
FLIPAdminAccess-893493035022 profile (override via LZA_AWS_PROFILE), keeps
the prod-grade TF_VAR_environment=prod hardening and origin/main deploy ref,
and namespaces trust kits as trust/.env.<CODE>.lza-prod so the
parallel-running legacy prod kits are never touched. Legacy PROD=true/stag
recipes and exported TF_VARs are byte-identical to before.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
feat(deploy/AWS): conditional network layer for the LZA platform-mana…
e317c47 
…ged VPC (#749)

When TF_VAR_lza_managed_network is set (PROD=lza) the VPC module is emptied
via its create flag and the AWSAccelerator VPC + subnets are discovered by
Name tag instead (new network_lza.tf), with every consumer rewired through
local.vpc_id / local.vpc_cidr_block / local.app_subnet_ids /
local.data_subnet_ids. Subnet placement on LZA: RDS goes to the isolated
data subnets; ECS tasks, ALB, RDS Proxy, EFS mount targets and EC2 hosts go
to the TGW-routed app subnets. The in-account VPC endpoints, DHCP options
and /flip/networking/* SSM params (legacy TGW coupling) are gated off there
— all SCP-denied or platform-provided. moved blocks keep legacy state
aligned where a count was added; on legacy envs the flag is false and the
resolved config is unchanged.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
feat(deploy/AWS): gate the SG-drift stack and public FL NLB off on LZA (
65f052b 
#749)

On the LZA account the org security baseline (Control Tower org trail,
GuardDuty, Security Hub, Config) owns change monitoring, so FLIP's own
CloudTrail->EventBridge->Lambda SG-drift plumbing is gated off there; and
with no IGW + VPC Block Public Access an internet-facing NLB is impossible
in-account, so the FL-server NLB, its target group, DNS record, SG rules and
the ECS load_balancer wiring are gated off too (the fl-server/fl-api
services themselves stay deployable) until the WP2 FL-inbound decision.
Adds the LZA-only ECR pull-through execution-role grant (ghcr/ +
ecr-public/ cache prefixes) and makes the Docker-Hub EFS-provision utility
image env-overridable so LZA can pull it through the credential-less
ecr-public cache. Legacy envs resolve to identical config; moved blocks
cover every added count.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
feat(deploy/AWS): support a DNS-less first bring-up on the default Cl…
84a3b48 
…oudFront domain (#749)

New manage_dns toggle (MANAGE_DNS in the env file; unset => true so legacy
prod/stag are untouched). false — the zone-less first LZA bring-up, before
the platform-side DNS migration lands — skips the Route53 zone lookup, every
record, and both DNS-validated ACM certs: CloudFront serves *.cloudfront.net
on the default viewer cert (no aliases), the CF->ALB VPC-origin leg falls
back to plain HTTP on the same listener (an ALB HTTPS listener needs an
ISSUED cert, which needs DNS validation), and the bucket CORS + Cognito
URLs follow the CloudFront default domain via local.ui_origin so uploads,
downloads and sign-in work before any DNS exists. Listener keys and the
/api/* rule attachment are unchanged in both modes to keep state stable;
moved blocks cover the counts added to certs and records.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): document the PROD=lza deployment path (#749)
c2d2a97 
New README section "Deploying to the LZA account (FLIPProduction)":
prerequisites (SSO profile, out-of-band state bucket + ECR pull-through
cache rules), what PROD=lza selects, platform-managed vs FLIP-managed
split, the .env.lza-prod value deltas (flip-lza-* buckets, pull-through
registry, MANAGE_DNS), the DNS-less first bring-up on the default
CloudFront domain with its limitations, and what is not yet exercised on
LZA. CLAUDE.md + AGENTS.md updated in lockstep (profile row, network_lza.tf
table entry, deploy-commands pointer); root CLAUDE.md PROD values extended.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
feat(deploy/AWS): route the Flower register-keys one-shot through the…
8dcdf5a 
… LZA-aware subnet locals (#749)

ecs_flower.tf landed on develop after the LZA branch was cut and still read
module.flip_vpc.private_subnets directly — an empty list on the LZA account,
where the VPC module is emptied by create_vpc = false. local.app_subnet_ids
resolves to exactly module.flip_vpc.private_subnets on the legacy path, so
this is a no-op for prod/stag.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): refresh the LZA state notes — multi-AZ landed, FL i…
71e13ab 
…ngress proven (#749)

The multi-AZ -b subnets landed via lza#38 (ALB/RDS unblocked, with the
TGW-attachment AZ caveat reported on PR#830), and the FL ingress chain was
proven end-to-end through the networking-account edge NLB (FLIP#829).

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur atriaybagur linked an issue Aug 17, 2026 that may be closed by this pull request
Migrate FLIP AWS deployment to the LZA FLIPProduction account #749
Open
44 tasks
@github-actions github-actions Bot changed the title feat(deploy/AWS): env-gated PROD=lza deployment path for the LZA FLIPProduction account (#749 WP1) Migrate FLIP AWS deployment to the LZA FLIPProduction account Aug 17, 2026
@atriaybagur atriaybagur self-assigned this Aug 17, 2026
@atriaybagur atriaybagur changed the title Migrate FLIP AWS deployment to the LZA FLIPProduction account feat(deploy/AWS): env-gated PROD=lza deployment path for the LZA FLIPProduction account (#749 WP1) Aug 17, 2026
@atriaybagur
refactor(deploy/AWS): short 'lza-prod' profile alias for the LZA guar…
db86cfe 
…d, matching prod/stag (#749)

Replaces the auto-generated FLIPAdminAccess-893493035022 profile name with
the same short-alias convention the other envs use (override still via
LZA_AWS_PROFILE). Also corrects the stale FlipDeveloperAccess-080369786334
row in the CLAUDE/AGENTS profile table — that permission set on the legacy
staging account was never a distinct environment; the dev row now documents
the dev/ root's profile. Inert for PROD=true/stag: LZA_AWS_PROFILE is only
read in the PROD=lza branch of the guard.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): correct the multi-AZ state — data-b and tgw-b subne…
5a9343f 
…ts still missing (#749)

Verified against the live FLIPProduction account: prod-app-b exists (ALB
unblocked) but prod-data-b does not (RDS subnet group still fails >=2-AZ)
and prod-tgw-b does not (TGW attachment single-subnet, AZ-b blackhole).

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(flip-api): put src/ on PYTHONPATH now the boot-time sync is gone
52bfb5c 
UV_NO_SYNC exposed that nothing ever baked the flip_api package into the
venv: the Dockerfile syncs dependencies before COPY src/, and the
boot-time 'uv run' re-sync (the PyPI fetch the previous commit removed)
was what installed the project editable on every start. PYTHONPATH makes
the src layout importable directly -- one mechanism for prod images and
dev bind-mounts alike, with no boot-time resolution.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
atriaybagur and others added 10 commits August 20, 2026 10:36
@atriaybagur
fix(deploy/AWS): classic hosted UI on LZA -- Managed Login blocks Pri…
f05c8e6 
…vateLink

Cognito refuses interface-endpoint (PrivateLink) API access to user pools
with Managed Login (v2) configured, and the egress-less LZA account's only
route to cognito-idp is the central interface endpoint -- every boot-time
Cognito call failed with InvalidParameterException and flip-api could not
start. The hosted-UI generation is cosmetic for FLIP (the UI signs in via
the SDK), so the LZA mode runs the classic hosted UI (v1); the
single-account mode keeps Managed Login (v2), unchanged.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(deploy/AWS): pin the LZA user pool to the LITE tier
6aa2e5e 
The v1-domain downgrade wasn't enough: new pools default to the
ESSENTIALS tier, and the tier itself is what Cognito's PrivateLink gate
reads as 'ManagedLogin configured' -- flip-api's startup
DescribeUserPoolClient kept failing through the interface endpoint even
with the classic hosted UI. LITE matches what the grandfathered legacy
pools effectively run and covers every flow FLIP uses (SDK sign-in, TOTP
MFA, admin APIs). Null on legacy: the argument is never sent, so
existing pools keep their tier and the legacy plan is unchanged.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(deploy/AWS): no hosted-UI domain on LZA -- it latches the Private…
0f41212 
…Link block

Empirical result (2026-08-20): a user pool domain that has ever had
Managed Login keeps the pool in 'ManagedLogin configured' through an
in-place downgrade to v1 -- ListUsers via the interface endpoint kept
returning the ManagedLogin refusal with the domain present at LITE+v1
and returned 200 the moment the domain was deleted. FLIP never uses the
hosted UI (SDK sign-in), so the LZA mode simply creates no domain;
legacy keeps domain + Managed Login v2, unchanged (moved block covers
the count).

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(deploy/AWS): admit the VPC CIDR on the LZA ALB web rule
16c0edc 
The FL NLB SG already concats the VPC CIDR with the networking ingress
CIDRs; the ALB rule only admitted the relay path, so in-VPC callers
(probe verification, internal consumers) timed out against a healthy
ALB. Same shape on both now, and the rule no longer vanishes when the
ingress CIDR list is empty.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(deploy/AWS): admit the internal FL NLB onto the fl-server task SG
f647cd4 
The legacy nlb-grpc admission rule references the legacy NLB's SG and is
gated off on LZA, leaving the fl-server task with no ingress on its FL
port -- the internal NLB's health checks could never pass and ECS churned
unhealthy targets. Mirror the legacy pattern against the internal NLB's
SG, LZA-gated like the rest of fl_ingress_lza.tf.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
Merge develop into 749-lza-terraform-env (pick up the MFA first-login…
ac31270 
… cache fix, #1016)

Signed-off-by: at24_bioeng625-pc <triayalex@gmail.com>
@atriaybagur
fix(lza): pin fl-server-net-1 to the internal FL NLB static IPs for f…
2dcbaa1 
…l-api

The NVFLARE admin kit targets the bare host fl-server-net-1 (a SAN on the
server cert). Legacy resolves it via the flip.local DHCP search domain
(dhcp.tf), which the LZA-managed VPC cannot carry - Fargate offers no search
domains - so fl-api's admin session died at boot on LZA (ClientConnectorDNSError),
the app failed its lifespan startup, and the hub saw connection-refused on
every net-status call (FL nets card never loaded). extraHosts pins the name
to the internal FL NLB's deterministic subnet_mapping IPs; the merge is empty
on legacy so its container definition stays byte-identical.

Signed-off-by: at24_bioeng625-pc <triayalex@gmail.com>
@atriaybagur
Revert "fix(lza): pin fl-server-net-1 to the internal FL NLB static I…
fe6d935 
…Ps for fl-api"

This reverts commit 2dcbaa1.
@atriaybagur
fix(lza): resolve the admin kit's bare fl-server-net-1 via a private …
646a3f8 
…hosted zone

Fargate's awsvpc network mode rejects extraHosts (the first attempt,
reverted), so publish the bare name the NVFLARE admin kit targets as a
single-label private hosted zone on the workload VPC, apex A records
pointing at the internal FL NLB's deterministic static IPs. Legacy keeps
resolving it via the flip.local DHCP search domain; the zone is LZA-gated.

Signed-off-by: at24_bioeng625-pc <triayalex@gmail.com>
@atriaybagur
fix(lza): codify the accelerator-managed ALB access logging
07163fa 
The LZA guardrail enables ALB access logging to the LogArchive account's
central bucket out-of-band, so every plan proposed reverting enabled=true.
Mirror it behind var.lza_elb_access_logs_bucket (empty default keeps the
access_logs block omitted and legacy plans byte-identical).

Signed-off-by: at24_bioeng625-pc <triayalex@gmail.com>
Copilot AI reviewed Aug 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an environment-gated PROD=lza deployment mode in deploy/providers/AWS to support deploying FLIP into the LZA-provisioned FLIPProduction AWS account while keeping legacy stag / prod paths functionally unchanged. This introduces an LZA “platform-managed network” axis (data-sourced VPC/subnets, no in-account endpoints/DHCP/VPC-origin CloudFront), plus the LZA ingress handoff plumbing needed for the two-tier edge design, and a small flip-api boot change to avoid boot-time uv dependency resolution in an egress-less environment.

Changes:

  • Add PROD=lza Makefile + Terraform wiring (TF_VAR_environment=prod plus TF_VAR_lza_managed_network=true) and related docs/scripts updates.
  • Implement LZA-specific infrastructure adaptations (network data lookups, internal FL NLB with static IPs + SSM handoff params, LZA-specific gating for CloudFront/VPC endpoints/DHCP/SG-drift stack).
  • Prevent flip-api crash-loop on egress-less boot by disabling boot-time uv sync and setting PYTHONPATH to the src/ layout.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
flip-api/entrypoint.sh Disables boot-time uv sync and sets PYTHONPATH for src-layout imports (needed for egress-less LZA boot + dev mounts).
deploy/providers/AWS/vpc_endpoints.tf Gates VPC endpoints off for LZA-managed network; switches to shared network locals.
deploy/providers/AWS/variables.tf Adds LZA-specific variables (managed network toggle, VPC discovery name, edge distribution inputs, etc.).
deploy/providers/AWS/services.tf Uses local.ui_origin for bucket CORS; adapts Cognito module inputs for LZA (hosted-UI/Managed Login constraints).
deploy/providers/AWS/service_discovery.tf Uses local.vpc_id to support both created and data-sourced VPCs.
deploy/providers/AWS/security.tf Gates SG-drift monitoring stack off on LZA; adds state-migration moved blocks for new counts.
deploy/providers/AWS/scripts/register-trusts.sh Adds PROD=lza kit suffix selection (.lza-prod).
deploy/providers/AWS/s3_logging.tf Allows overriding globally-unique log bucket names (needed for LZA parallel prod).
deploy/providers/AWS/README.md Documents PROD=lza behavior and LZA deployment prerequisites/constraints.
deploy/providers/AWS/rds_proxy.tf Places RDS Proxy into app subnets via LZA-aware locals.
deploy/providers/AWS/parameter_store.tf Gates legacy TGW-coupling SSM params off on LZA; adds moved blocks for new counts.
deploy/providers/AWS/network_lza.tf Adds LZA VPC/subnet discovery and defines local.vpc_id / local.app_subnet_ids / local.data_subnet_ids.
deploy/providers/AWS/modules/cognito/variables.tf Adds inputs to control hosted UI domain creation + managed login version/tier.
deploy/providers/AWS/modules/cognito/outputs.tf Adjusts module outputs for optional hosted-UI domain.
deploy/providers/AWS/modules/cognito/main.tf Gates hosted UI domain for LZA; adds tier/login version controls.
deploy/providers/AWS/Makefile Adds PROD=lza env file selection, AWS profile guard, TF var mapping, and LZA-specific deploy-ui behavior.
deploy/providers/AWS/main.tf Makes VPC creation conditional for LZA; routes VPC/subnet usage via locals; adjusts LB/DNS resources for manage_dns + LZA.
deploy/providers/AWS/iam_ecs.tf Adds LZA-only ECR pull-through cache import permissions to the ECS execution role.
deploy/providers/AWS/fl_ingress_lza.tf Adds LZA-only ingress plumbing: internal FL NLB w/ static IPs, ALB ingress from networking relay, and /flip/networking/* SSM handoff params.
deploy/providers/AWS/efs.tf Places EFS mount targets in app subnets via LZA-aware locals.
deploy/providers/AWS/ecs_sg.tf Gates legacy NLB→fl-server ingress rule off on LZA; uses LZA-aware VPC CIDR local.
deploy/providers/AWS/ecs_services.tf Uses app subnets via locals; switches fl-server LB TG between legacy and LZA internal NLB.
deploy/providers/AWS/ecs_flower.tf Uses LZA-aware subnets for the Flower supernode key-registration task networking config.
deploy/providers/AWS/ecs_efs_provision.tf Uses LZA-aware subnets and makes the EFS provision image configurable (for egress-less LZA).
deploy/providers/AWS/dhcp.tf Gates DHCP options association off on LZA-managed VPC.
deploy/providers/AWS/cloudfront.tf Gates VPC origin + workload distribution off on LZA; adapts behavior for manage_dns; adds local.ui_origin derivation and LZA edge policy wiring.
deploy/providers/AWS/certificate.tf Gates DNS-validated ACM flow off when manage_dns=false.
deploy/providers/AWS/CLAUDE.md Updates AWS deploy agent docs for new LZA files/commands.
deploy/providers/AWS/AGENTS.md Mirrors AWS deploy agent docs for new LZA files/commands.
CLAUDE.md Documents PROD=lza as an environment option in root instructions.
AGENTS.md Mirrors root instructions change documenting PROD=lza.
.gitignore Adds .env.lza-prod to ignored env files.
Suppressed comments (1)

deploy/providers/AWS/cloudfront.tf:1155

  • aws_cloudfront_distribution.flip_ui is count = 0 on LZA, so one(aws_cloudfront_distribution.flip_ui[*].domain_name) becomes one([]) and errors during plan/apply. Guard this output so LZA plans can succeed (e.g., return null when not created).
output "CloudfrontDistributionDomain" {
  description = "CloudFront distribution CloudFront-assigned domain (*.cloudfront.net). Use for pre-cutover smoke tests."
  value       = one(aws_cloudfront_distribution.flip_ui[*].domain_name)
}

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deploy/providers/AWS/cloudfront.tf
Comment thread deploy/providers/AWS/variables.tf
Comment thread deploy/providers/AWS/modules/cognito/outputs.tf
@atriaybagur
fix(deploy/AWS): access_logs 'off' is null, not {} -- empty map broke…
f53e81c 
… every legacy plan

The ALB module types access_logs as an object with a required bucket
attribute (default null), so the {} passed on the legacy branch of the
conditional failed type conversion and errored the whole prod/stag plan.
Caught by a live stag plan from this branch; with null it plans clean --
only develop-ahead drift and the out-of-band ECS Exec enables remain,
and the sg-drift moved blocks land as pure renames.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): document the two-phase LZA edge wiring; refresh WP1…
0d6af84 
…-era state notes

The lza_web_edge_domain/_arn variables deliberately carry no
required-when-LZA validation: the networking-account edge stack is built
from this stack's /flip/networking/* handoff outputs, so the first
workload apply necessarily precedes the edge distribution -- empty values
mean a grants-nobody bucket policy (fail-closed) and a placeholder
ui_origin until the wiring re-apply. Also refresh the LZA section lines
superseded by the WP3 commits on this branch (FL ingress is wired, the
workload CloudFront is gated off in favour of the edge distribution).

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(fl-api-flower): drive the flwr CLI from the baked venv, not uvx
7cd6b5b 
uvx resolves its tool environment from PyPI at call time, so on the
egress-less LZA account (FLIP#749) every flwr invocation -- run, list,
stop, federation list -- timed out against pypi.org: submissions errored
the model and status/reconcile went dark. flwr is already a direct
dependency of this service's venv (on PATH in the image), so invoke that
binary directly; no network at call time on any environment.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
feat(deploy/AWS): disable Flower runtime dependency installs on LZA
eb12a65 
The SuperLink's per-run 'uv sync' resolves the app env from package
indexes at run time -- unreachable from the egress-less account, so every
ServerApp died on its first fetch (hatchling, flip-utils' build backend;
Flower exit code 608). Flower's documented restricted-network mode is to
preinstall app dependencies and disable runtime installation; fl-base
already bakes the full ML env, so LZA sets
FLWR_DISABLE_RUNTIME_DEPENDENCY_INSTALLATION=1 on the SuperLink task.
Researcher package additions there ship via image rebuild, not
pyproject. Legacy envs keep runtime installs (merge with {} keeps the
map identical).

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): frame the two deployment modes; refresh LZA state n…
9946924 
…otes

Present the AWS Terraform root as offering two permanently supported
deployment modes rather than a migration in progress: the self-contained
single-account shape (PROD=stag/true, FLIP creates its own VPC and edge)
and the platform-managed shape (PROD=lza) for a Landing Zone Accelerator
estate. Both docs previously implied the first was on its way out.

Correct two stale caveats in the LZA section that the same PR contradicts:

- multi-AZ has landed platform-side (both AZs carry app/data/tgw subnets),
  so the RDS two-AZ constraint and the -b blackhole no longer apply, and
  no FLIP-side change was needed since the subnet lookups glob Name tags;
- FL ingress is wired — fl_ingress_lza.tf creates the internal NLB, its
  target group fronting fl-server-net-1, and the /flip/networking/* SSM
  handoff the networking account's edge consumes.

The auxiliary-tooling caveat (full-deploy chains, check_status.py,
update_env.py, destroy) is unverified against PROD=lza and stays.

Also drop the workload account ID from tracked files — the account is
identified by its `lza-prod` profile alias, matching how prod/stag are
referenced — and rename the section anchor accordingly.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): confirm the multi-AZ state against the live account
1eae093 
Re-checked today rather than resting on the 2026-08-20 notes: all six
platform subnets exist (app/data/tgw in both AZs), flip-db-subnet-group
spans eu-west-2a and eu-west-2b, and the TGW attachment rides tgw-a and
tgw-b — so the note can state the RDS group and the attachment as facts
rather than as a requirement that is merely satisfiable.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs: cover both Central Hub deployment modes in the published guide
a74c091 
The Sphinx deployment guide described only the self-contained shape — own
VPC with public and private subnets, in-account CloudFront, a public FL
NLB, .env.stag/.env.production — every line of which is false under
PROD=lza. A reader of the published docs had no way to know a second mode
exists.

Adds a "Deployment modes" section contrasting the two, marks the
architecture section as describing the self-contained shape and summarises
what moves under a platform-managed one, and threads PROD=lza through the
prerequisites, the environment-file mapping and the key generation note. A
note records that full-deploy, status and destroy are the self-contained
chain and are not yet exercised on PROD=lza.

Operator detail deliberately stays in deploy/providers/AWS/README.md; this
page says the mode exists, what it implies, and where to read further.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
feat(deploy/AWS): PROD=lza-stag — staging on a platform-managed estate
ef6456e 
Generalises the LZA mode from a single environment into a family. The two
axes were already orthogonal (environment vs lza_managed_network), so no
Terraform changes are needed — lza-stag composes environment=stag with the
platform-managed network:

- env file .env.lza-stag (gitignored), profile guard lza-stag
  (LZA_STAG_AWS_PROFILE), kit suffix .lza-stag, origin/develop deploys,
  and the prod-only RDS hardening off, all matching legacy stag semantics;
- LZA_VPC_NAME is mandatory on lza-stag: the variables.tf default names
  the prod VPC template, and there is no defaultable staging name, so the
  Makefile refuses to run rather than letting the data lookup fail with an
  opaque "no results";
- the deploy-ui invalidation skip now covers both LZA values.

Docs follow: the two-mode table lists both values, a comparison table
contrasts lza vs lza-stag (including the one-account-per-environment rule
— the stack's resource names and the /flip/networking/* handoff params
are fixed per account by design), and the Sphinx guide mentions the
staging value in the same breath as lza.

Verified: guards exercised for wrong-profile, missing-LZA_VPC_NAME and
missing-env-file; derived values probed for all four PROD values
(lza-stag → environment=stag, lza_managed_network=true, kit_suffix
lza-stag, origin/develop); `make -n plan` byte-identical to HEAD for
PROD=stag and PROD=lza; terraform fmt and the docs build clean.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
Merge branch 'develop' into 749-lza-terraform-env
4c81a70 
One conflict, in the root CLAUDE.md/AGENTS.md pair: this branch extends the
`PROD` bullet with the `lza` / `lza-stag` values, while #958 (FLIP_INSTANCE)
added three new bullets — `FLIP_INSTANCE`, `MAIN_ENV_FILE`, `DB_PORT` —
immediately after the old one-line `PROD` bullet. Resolved as the union: the
LZA `PROD` line followed by develop's three bullets. No wording change was
needed for `MAIN_ENV_FILE`: its root-Makefile default only ever covers the
self-contained environments, and the LZA env files (`.env.lza-prod`,
`.env.lza-stag`) are still selected by deploy/providers/AWS/Makefile, which is
the only place `PROD=lza*` is meaningful. AGENTS.md regenerated from CLAUDE.md.

Everything else auto-merged; the only file both sides touched under
deploy/providers/AWS is README.md, whose LZA section survives intact.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(deploy/AWS): teach register-trusts.sh and add_fl_kits.sh the LZA …
84a577a 
…PROD values

PROD=lza-stag (ef6456e) was only half-plumbed: the two scripts that re-derive
the environment from PROD on their own never learned the LZA values.

- register-trusts.sh mapped true/lza/* — so PROD=lza-stag wrote `.stag` kits
  while the deploy Makefile's KIT_ENV_SUFFIX (and every trust-deploy target
  behind it) reads `.lza-stag`: registration "succeeded" and the next target
  failed with kit-not-found. Add the lza-stag case; keep the two in lockstep.
- add_fl_kits.sh refused anything but stag|true, so the documented pool-growth
  path was dead on BOTH LZA values. The LZA pair now rides the legacy
  workspaces / project YAMLs on purpose: their kits carry the same server FQDN
  and root CA so trusts keep their client kits across the DNS cutover (FLIP#749);
  a staging estate provisioned with a fresh CA trips the existing fingerprint
  guard instead of minting silently.

test_add_fl_kits.sh gains a PROD_MODE knob and covers lza-stag → stag env,
lza → prod env, and an unknown value being refused before anything is touched.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(fl-services): load .env.lza-prod / .env.lza-stag for the kit-uplo…
ac1951c 
…ad targets

`make -C fl-services/<backend> upload-kits-to-s3 PROD=lza-stag` (and Flower's
upload-creds-to-s3) fell through the true|stag env-file chain to
.env.development — the dev bucket, with no profile guard and no error. Add
the two LZA branches, mirroring the deploy Makefile, and treat lza like true
for FL_KIT_WORKSPACE (the prod workspace — same FQDN + CA by design, FLIP#749).

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(make): name LZA trust kits .lza-prod / .lza-stag from the root ki…
d5d74b6 
…t-file targets

`make new-trust … PROD=lza-stag` — the exact hint register-trusts.sh prints —
scaffolded trust/.env.<CODE>.development, and sync-trust-kit(s) read
.env.development, because the root Makefile's ENV / MAIN_ENV_FILE only knew
true|stag. Both now follow the deploy Makefile's KIT_ENV_SUFFIX (.lza-prod /
.lza-stag), and new_trust.py accepts the two suffixes and hints the PROD flag
that reads them back (lza-prod → PROD=lza). ENV is only ever a kit-file token
at the root — the composes hardcode their own ENV — so nothing else moves.

CLAUDE.md (+ the AGENTS.md twin): the PROD bullet now names the kit-file and
FL-kit-upload targets the LZA values reach, and MAIN_ENV_FILE's default list
carries the two LZA files.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
docs(deploy/AWS): list the raw TF_VAR exports an LZA env file carries
e6cda87 
The .env.lza-prod block omitted the four Terraform inputs that have no
Makefile mapping and ride as `export TF_VAR_…=` lines (ingress-VPC CIDRs,
LogArchive ELB-logs bucket, edge domain + distribution ARN), so a staging
env file could not be built from the doc alone. List them with placeholders,
and say which are estate-wide on staging and which stay empty until a
staging edge exists.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
Merge branch 'develop' into 749-lza-terraform-env (pick up the checko…
852590d 
…v lint, #1057)

Conflicts were the #1057 `# checkov:skip=` annotations landing on the same
lines as this branch's LZA gating — main.tf (fl_server_nlb: keep develop's
module version pin + the LZA `create` flag), parameter_store.tf (both legacy
TGW-handoff params: skip comment + `count`), cloudfront.tf (flip_ui: skip
comment + `count`). Resolved as the union; `terraform fmt`/`validate` clean.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
fix(deploy/AWS): satisfy the checkov lint on fl_ingress_lza.tf
e1ee31e 
develop's checkov-lint job (#1057) now scans the LZA files for the first time:

- CKV_TF_2: pin module.fl_server_internal_nlb to `~> 10.0` like its legacy
  twin — the resolved version is already 10.5.0, so no change on apply.
- CKV2_AWS_34 x4: the /flip/networking/* handoff parameters are non-secret
  values read cross-account by the networking account's edge stack, where an
  AWS-managed CMK could not be decrypted — the same rationale develop applied
  to the legacy TGW handoff params. Acknowledged in-code, not by weakening
  the check list.

Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@martinchapman martinchapman Awaiting requested review from martinchapman

At least 1 approving review is required to merge this pull request.

Assignees

@martinchapman martinchapman

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Migrate FLIP AWS deployment to the LZA FLIPProduction account

3 participants

@atriaybagur @martinchapman