diff --git a/.gitignore b/.gitignore index 3384d12d7..c44a632e4 100644 --- a/.gitignore +++ b/.gitignore @@ -456,16 +456,15 @@ flip-utils/dist/ # version control (covered by .claude/* above, listed explicitly for clarity). .claude/worktrees/ -# fl-tutorials generated / uncommitted artifacts (data/, the spleen-seg uv.lock), scoped under the -# per-backend tutorial trees (fl-tutorials//) so a bare `data/` doesn't over-match -# other data/ dirs in the monorepo: -# - generated tutorial datasets (spleen/xray images + accession CSVs) -# - tutorial uv.lock (tutorials intentionally don't pin a strict lockfile) -fl-tutorials/nvflare/data/ -fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/uv.lock +# fl-tutorials generated / uncommitted artifacts, scoped (not a bare `data/`, which would +# over-match other data/ dirs in the monorepo): +# - the shared tutorial-dataset root both backends' harnesses read (fl-tutorials/datasets/ +# downloads land here: spleen/xray/arkplus images + accession CSVs + the HF cache) +# - the dataset tooling's uv.lock (intentionally not pinned to a strict lockfile) +fl-tutorials/data/ +fl-tutorials/datasets/spleen/uv.lock fl-services/flower/provision/creds/ fl-services/flower/runs/ -fl-tutorials/flower/data/ # Round-timing extraction output (scripts/fl_round_metrics/ extractors and the # tutorials' `make round-metrics` write diff --git a/AGENTS.md b/AGENTS.md index 642db8539..2e45adf89 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ FLIP/ ├── flip-utils/ # FLIP Python library (pip-installable flip-utils) ├── fl-services/ # FL Docker services + network provisioning, per backend (Makefile owns build/provision/up/down/submit; flower also up-secure): fl-services/nvflare/{fl-base,fl-server,fl-client,fl-api-base, provision/{net-*_project_*.yml, scripts/, workspace-{dev,stag,prod}/ gitignored}}, fl-services/flower/{fl-base,superlink,supernode,fl-api-flower, provision/{scripts/, creds/ gitignored}} (#622) ├── fl-apps/ # FL app templates per backend: fl-apps/nvflare/{standard,evaluation,diffusion_model,fed_opt} (all Client-API), fl-apps/flower/{standard,evaluation} + check_required_files.sh (cross-backend CI validator at root) -├── fl-tutorials/ # FL tutorials per backend (all NVFLARE ones are Client-API apps): fl-tutorials/nvflare/{image_*}, fl-tutorials/flower/{xray_classification,3d_spleen_segmentation*} (root Makefile forwards by FL_BACKEND); xray classification, spleen seg/eval, diffusion. Plus fl-tutorials/tests/ — CPU-only pytest over the tutorial transform chains (#871) plus a static `min_clients` wiring guard covering fl-apps/flower too, run by `make -C fl-tutorials test` +├── fl-tutorials/ # FL tutorials per backend (all NVFLARE ones are Client-API apps): fl-tutorials/nvflare/{image_*}, fl-tutorials/flower/{xray_classification,3d_spleen_segmentation*} (root Makefile forwards by FL_BACKEND); xray classification, spleen seg/eval, diffusion. Shared dataset tooling in fl-tutorials/datasets/ (download/derive/enrich, single copy for both backends — the download-*-data + upload-spleen-labels targets), outputs in the shared gitignored fl-tutorials/data/. Plus fl-tutorials/tests/ — CPU-only pytest over the tutorial transform chains (#871) plus a static `min_clients` wiring guard covering fl-apps/flower too, run by `make -C fl-tutorials test` ├── map-apps/ # MONAI Application Package (MAP) templates for packaging FLIP-trained models for clinical deployment ├── trust/ │ ├── trust-api/ # Trust API gateway (Python/FastAPI) @@ -159,7 +159,7 @@ cause. (Older app copies die with torch's opaque `num_samples=0` instead.) `e2e_smoke` has a hook for exactly this: `--data-enrichment-cwd` + `--data-enrichment-cmd` run a shell command **between the image pull and training**, with `FLIP_PROJECT_ID` exported. Since FLIP#776 the spleen uploader is -in-tree at `fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/upload_spleen_labels_to_xnat.py` +in-tree at `fl-tutorials/datasets/spleen/upload_spleen_labels_to_xnat.py` (the shared dataset-tooling tree) — **no private repo required**. It resolves each trust's XNAT project by `secondary_ID == `, fetches the accession→MSD-case mapping at run time from the public `aicentreflip/trust-data` dataset (`omop-csv//spleen_project/image_occurrence.csv`, which also carries `source_trust`), and writes each @@ -198,8 +198,8 @@ instead: cd flip-api && uv run python -m tests.e2e_smoke \ --model-files-dir ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/app_files \ --query-file ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/query.sql \ - --data-enrichment-cwd ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation \ - --data-enrichment-cmd 'uv run --no-project --with ../../../../flip-utils python utils/upload_spleen_labels_to_xnat.py --flip-project-id "$FLIP_PROJECT_ID" --labels-dir ../../data/spleen/images --xnat-url http://127.0.0.1:8104 --xnat-url http://127.0.0.1:8106' + --data-enrichment-cwd ../fl-tutorials/datasets/spleen \ + --data-enrichment-cmd 'uv run --no-project --with ../../../flip-utils python upload_spleen_labels_to_xnat.py --flip-project-id "$FLIP_PROJECT_ID" --labels-dir ../../data/spleen/images --xnat-url http://127.0.0.1:8104 --xnat-url http://127.0.0.1:8106' ``` (The `$`-escaping trap still applies to any hand-written `EXTRA_ARGS`: `make -C flip-api …` expands once, so @@ -209,7 +209,7 @@ there. The in-Makefile targets handle this for you. Note that `e2e_smoke_spleen` tutorial (`../fl-tutorials/flower/3d_spleen_segmentation/{app,query.sql}`), not the NVFLARE paths shown in the direct-invocation example above — pair it with `FL_BACKEND=flower`, or invoke `tests.e2e_smoke` directly with the NVFLARE paths for the NVFLARE tutorial. The enrichment step itself is backend-agnostic and runs out of the -NVFLARE tree either way.) +shared `fl-tutorials/datasets/spleen/` tree either way.) Enrichment must land **after** the pull and after DICOM→NIfTI conversion; the hook's position guarantees that. The uploader derives each target filename from the converted `input_*.nii.gz`, so with no `NIFTI` resource it diff --git a/CLAUDE.md b/CLAUDE.md index 4cbd34632..72181780c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,7 +16,7 @@ FLIP/ ├── flip-utils/ # FLIP Python library (pip-installable flip-utils) ├── fl-services/ # FL Docker services + network provisioning, per backend (Makefile owns build/provision/up/down/submit; flower also up-secure): fl-services/nvflare/{fl-base,fl-server,fl-client,fl-api-base, provision/{net-*_project_*.yml, scripts/, workspace-{dev,stag,prod}/ gitignored}}, fl-services/flower/{fl-base,superlink,supernode,fl-api-flower, provision/{scripts/, creds/ gitignored}} (#622) ├── fl-apps/ # FL app templates per backend: fl-apps/nvflare/{standard,evaluation,diffusion_model,fed_opt} (all Client-API), fl-apps/flower/{standard,evaluation} + check_required_files.sh (cross-backend CI validator at root) -├── fl-tutorials/ # FL tutorials per backend (all NVFLARE ones are Client-API apps): fl-tutorials/nvflare/{image_*}, fl-tutorials/flower/{xray_classification,3d_spleen_segmentation*} (root Makefile forwards by FL_BACKEND); xray classification, spleen seg/eval, diffusion. Plus fl-tutorials/tests/ — CPU-only pytest over the tutorial transform chains (#871) plus a static `min_clients` wiring guard covering fl-apps/flower too, run by `make -C fl-tutorials test` +├── fl-tutorials/ # FL tutorials per backend (all NVFLARE ones are Client-API apps): fl-tutorials/nvflare/{image_*}, fl-tutorials/flower/{xray_classification,3d_spleen_segmentation*} (root Makefile forwards by FL_BACKEND); xray classification, spleen seg/eval, diffusion. Shared dataset tooling in fl-tutorials/datasets/ (download/derive/enrich, single copy for both backends — the download-*-data + upload-spleen-labels targets), outputs in the shared gitignored fl-tutorials/data/. Plus fl-tutorials/tests/ — CPU-only pytest over the tutorial transform chains (#871) plus a static `min_clients` wiring guard covering fl-apps/flower too, run by `make -C fl-tutorials test` ├── map-apps/ # MONAI Application Package (MAP) templates for packaging FLIP-trained models for clinical deployment ├── trust/ │ ├── trust-api/ # Trust API gateway (Python/FastAPI) @@ -159,7 +159,7 @@ cause. (Older app copies die with torch's opaque `num_samples=0` instead.) `e2e_smoke` has a hook for exactly this: `--data-enrichment-cwd` + `--data-enrichment-cmd` run a shell command **between the image pull and training**, with `FLIP_PROJECT_ID` exported. Since FLIP#776 the spleen uploader is -in-tree at `fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/upload_spleen_labels_to_xnat.py` +in-tree at `fl-tutorials/datasets/spleen/upload_spleen_labels_to_xnat.py` (the shared dataset-tooling tree) — **no private repo required**. It resolves each trust's XNAT project by `secondary_ID == `, fetches the accession→MSD-case mapping at run time from the public `aicentreflip/trust-data` dataset (`omop-csv//spleen_project/image_occurrence.csv`, which also carries `source_trust`), and writes each @@ -198,8 +198,8 @@ instead: cd flip-api && uv run python -m tests.e2e_smoke \ --model-files-dir ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/app_files \ --query-file ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/query.sql \ - --data-enrichment-cwd ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation \ - --data-enrichment-cmd 'uv run --no-project --with ../../../../flip-utils python utils/upload_spleen_labels_to_xnat.py --flip-project-id "$FLIP_PROJECT_ID" --labels-dir ../../data/spleen/images --xnat-url http://127.0.0.1:8104 --xnat-url http://127.0.0.1:8106' + --data-enrichment-cwd ../fl-tutorials/datasets/spleen \ + --data-enrichment-cmd 'uv run --no-project --with ../../../flip-utils python upload_spleen_labels_to_xnat.py --flip-project-id "$FLIP_PROJECT_ID" --labels-dir ../../data/spleen/images --xnat-url http://127.0.0.1:8104 --xnat-url http://127.0.0.1:8106' ``` (The `$`-escaping trap still applies to any hand-written `EXTRA_ARGS`: `make -C flip-api …` expands once, so @@ -209,7 +209,7 @@ there. The in-Makefile targets handle this for you. Note that `e2e_smoke_spleen` tutorial (`../fl-tutorials/flower/3d_spleen_segmentation/{app,query.sql}`), not the NVFLARE paths shown in the direct-invocation example above — pair it with `FL_BACKEND=flower`, or invoke `tests.e2e_smoke` directly with the NVFLARE paths for the NVFLARE tutorial. The enrichment step itself is backend-agnostic and runs out of the -NVFLARE tree either way.) +shared `fl-tutorials/datasets/spleen/` tree either way.) Enrichment must land **after** the pull and after DICOM→NIfTI conversion; the hook's position guarantees that. The uploader derives each target filename from the converted `input_*.nii.gz`, so with no `NIFTI` resource it diff --git a/docs/source/user-guides/arkplus-fine-tuning.rst b/docs/source/user-guides/arkplus-fine-tuning.rst index 46b695be3..0df5a9dde 100644 --- a/docs/source/user-guides/arkplus-fine-tuning.rst +++ b/docs/source/user-guides/arkplus-fine-tuning.rst @@ -104,7 +104,7 @@ published reference dataset on Hugging Face: This pulls the site-1 / site-2 training splits from `aicentreflip/tutorials-arkplus-cxr-classification `_ -and lays them out under ``fl-tutorials/nvflare/data/arkplus/`` (gitignored). +and lays them out under ``fl-tutorials/data/arkplus/`` (gitignored). To use your own data instead, point the per-site ``.env.app`` values (``SITE{1,2}_IMAGES_DIR`` / ``SITE{1,2}_DATAFRAME``) at your directories. The trainer selects a diff --git a/fl-services/flower/README.md b/fl-services/flower/README.md index 9666e46ce..471bdbbc6 100644 --- a/fl-services/flower/README.md +++ b/fl-services/flower/README.md @@ -155,10 +155,10 @@ trains on an empty dataset with no signal that the data is missing. So download point both variables at it when bringing the net up: ```bash -make -C fl-tutorials download-xray-data FL_BACKEND=flower # → fl-tutorials/flower/data/xrays_mini_300/ +make -C fl-tutorials download-xray-data FL_BACKEND=flower # → fl-tutorials/data/xrays_mini_300/ make -C fl-services/flower up \ - DEV_IMAGES_DIR=$(pwd)/fl-tutorials/flower/data/xrays_mini_300/accession-resources \ - DEV_DATAFRAME=$(pwd)/fl-tutorials/flower/data/xrays_mini_300/sample_get_dataframe_response.csv + DEV_IMAGES_DIR=$(pwd)/fl-tutorials/data/xrays_mini_300/accession-resources \ + DEV_DATAFRAME=$(pwd)/fl-tutorials/data/xrays_mini_300/dataframe.csv ``` (Run from the repo root; `data/` is gitignored. `up-secure` takes the same two variables.) diff --git a/fl-tutorials/Makefile b/fl-tutorials/Makefile index 6615b484e..bae503f5d 100644 --- a/fl-tutorials/Makefile +++ b/fl-tutorials/Makefile @@ -9,22 +9,29 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Backend-aware entrypoint for the FL tutorial runners. The actual runner + harness live -# per-backend under fl-tutorials// (fl-tutorials/nvflare/ and fl-tutorials/flower/). This thin -# Makefile forwards every target to fl-tutorials/$(FL_BACKEND)/ so the documented entrypoint +# Backend-aware entrypoint for the FL tutorial runners. The runner + harness live per-backend +# under fl-tutorials// (fl-tutorials/nvflare/ and fl-tutorials/flower/); the shared +# dataset tooling lives in fl-tutorials/datasets/ (outputs in the gitignored fl-tutorials/data/, +# one download serving both backends). This thin Makefile forwards each target to the right +# sub-make so the documented entrypoint # make -C fl-tutorials [TUTORIAL=... NUM_CASES=...] # keeps working and selects the backend via FL_BACKEND (default nvflare). Command-line # variables are forwarded to the sub-make automatically. FL_BACKEND ?= nvflare -TUTORIAL_TARGETS := list-tutorials run-tutorial run-all-tutorials \ - download-xray-data download-spleen-data upload-spleen-labels \ +TUTORIAL_TARGETS := list-tutorials run-tutorial run-all-tutorials +# Dataset targets are backend-agnostic (shared data root, single copy in datasets/); +# FL_BACKEND still selects a per-harness variant where one exists (the two spleen downloads). +DATASET_TARGETS := download-xray-data download-spleen-data upload-spleen-labels \ download-arkplus-finetuning-data download-arkplus-eval-data -.PHONY: $(TUTORIAL_TARGETS) +.PHONY: $(TUTORIAL_TARGETS) $(DATASET_TARGETS) $(TUTORIAL_TARGETS): @$(MAKE) --no-print-directory -C $(FL_BACKEND) $@ +$(DATASET_TARGETS): + @$(MAKE) --no-print-directory -C datasets $@ FL_BACKEND=$(FL_BACKEND) + # --------------------------------------------------------------------------- # Lint / autoformat the tutorial sources. Runs ruff over the whole tree; ruff # auto-excludes the per-tutorial .venv/ trees and resolves config per-file diff --git a/fl-tutorials/README.md b/fl-tutorials/README.md index 89f1c92f8..5d7cc4593 100644 --- a/fl-tutorials/README.md +++ b/fl-tutorials/README.md @@ -47,9 +47,10 @@ make -C fl-tutorials run-tutorial TUTORIAL=xray_classification built `flare-fl-base` image and fail without it. The spleen examples take `download-spleen-data` in place of `download-xray-data`. -The simulator requires Docker, and GPU-backed examples require the NVIDIA Container Toolkit. Dataset downloads and -generated runs are kept in gitignored backend data/output directories. Run `make -C fl-tutorials run-all-tutorials` -only when you intentionally want the full, heavyweight suite. +The simulator requires Docker, and GPU-backed examples require the NVIDIA Container Toolkit. Dataset tooling is +shared across backends in [`datasets/`](datasets/) and downloads land in the shared gitignored `data/` root, so +one download serves both backends; generated runs stay in gitignored per-backend output directories. Run +`make -C fl-tutorials run-all-tutorials` only when you intentionally want the full, heavyweight suite. ## Check the sources without a GPU diff --git a/fl-tutorials/datasets/Makefile b/fl-tutorials/datasets/Makefile new file mode 100644 index 000000000..8405a13c1 --- /dev/null +++ b/fl-tutorials/datasets/Makefile @@ -0,0 +1,147 @@ +# Copyright (c) 2026 Guy's and St Thomas' NHS Foundation Trust & King's College London +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Shared dataset tooling for the FL tutorials. One download per dataset, consumed by BOTH +# backends' harnesses: datasets are backend-agnostic (a tutorial's NVFLARE and Flower twins +# train on the same data), so their download/derive/enrich tooling lives here, once, instead +# of duplicated per backend. The fl-tutorials root Makefile forwards these targets here, so +# the documented entrypoints keep working: +# +# make -C fl-tutorials download-xray-data +# make -C fl-tutorials download-spleen-data [NUM_CASES=<1-41>] [FL_BACKEND=flower] +# make -C fl-tutorials download-arkplus-finetuning-data +# make -C fl-tutorials download-arkplus-eval-data +# make -C fl-tutorials upload-spleen-labels FLIP_PROJECT_ID= # data enrichment +# +# Everything lands under the shared, gitignored fl-tutorials/data/ — see README.md here for +# the per-dataset layout. FL_BACKEND matters only where a dataset has per-harness variants +# (spleen: the NVFLARE simulator reads the raw MSD build, the Flower compose stack mounts the +# pre-built FLIP-format tree; both can coexist under data/spleen/). + +FL_BACKEND ?= nvflare +DATA_DIR := ../data +HF_CACHE := $(DATA_DIR)/.hf + +# Hugging Face dataset backing the xray tutorials and the FLIP-format spleen variant. +HF_TEST_DATA_REPO := aicentreflip/flip-fl-base-test-data +# Hugging Face dataset backing the three Ark+ tutorials (site{1,2} train + hold-out splits). +ARKPLUS_DATA_REPO := aicentreflip/tutorials-arkplus-cxr-classification +# Number of MSD spleen cases to download for the spleen tutorials (1-41). +NUM_CASES ?= 10 +# The MSD download/derive scripts run in spleen/'s own uv project (MONAI, pandas, natsort) — +# --project keeps the cwd here while resolving that env. +SPLEEN_UV := uv run --project spleen + +.PHONY: download-xray-data download-spleen-data download-spleen-msd-data \ + download-spleen-flip-format-data download-arkplus-finetuning-data \ + download-arkplus-eval-data upload-spleen-labels + +# Download the x-ray reference dataset from Hugging Face and normalise it into the layout +# both harnesses read (data/xrays_mini_300/, gitignored). Idempotent — safe to re-run. +# One canonical CSV name (dataframe.csv): the NVFLARE .env.app files and the Flower +# run-tutorial.sh both point at it. Script + README live in xrays_mini_300/ (see its README). +download-xray-data: + @uv run --no-project --with huggingface_hub python xrays_mini_300/download_xrays_dataset.py \ + --cache-dir $(HF_CACHE) \ + --output-dir $(DATA_DIR)/xrays_mini_300 \ + --repo-id $(HF_TEST_DATA_REPO) + +# The spleen dataset has two variants sharing data/spleen/: the raw MSD build (images/ + +# dataframe.csv — what the NVFLARE simulator reads, and where the enrichment labels come +# from) and the pre-built FLIP-format tree (accession-resources/ + +# sample_get_dataframe_response.csv — what the Flower compose stack mounts). The plain +# download-spleen-data entrypoint keeps its historical per-backend meaning; call the +# explicit variant targets directly to fetch a specific one. +ifeq ($(FL_BACKEND),flower) +download-spleen-data: download-spleen-flip-format-data +else +download-spleen-data: download-spleen-msd-data +endif + +# Download the MSD spleen dataset and build its accession CSV (large; NUM_CASES caps the +# subject count). Runs in spleen/'s own uv project for MONAI. +download-spleen-msd-data: + @echo "⬇️ Downloading MSD spleen dataset ($(NUM_CASES) cases) — large, runs in spleen/'s uv project..." + @$(SPLEEN_UV) python spleen/download_spleen_dataset.py --output_dir $(DATA_DIR)/spleen/images --num_cases $(NUM_CASES) + @$(SPLEEN_UV) python spleen/create_spleen_accession_csv.py --images_dir $(DATA_DIR)/spleen/images --output_csv $(DATA_DIR)/spleen/dataframe.csv + @echo "✅ MSD spleen data ready at $(abspath $(DATA_DIR))/spleen/ — make -C fl-tutorials run-tutorial TUTORIAL=3d_spleen_segmentation" + +# Download the pre-built FLIP-format spleen tree (fixed 6-case snapshot; ignores NUM_CASES) +# plus the evaluation-tutorial checkpoint. Removes only its own outputs, so an MSD build in +# the same data/spleen/ survives a re-download. A pure Hugging Face fetch, so it runs via +# `uv run --no-project --with huggingface_hub` like the xray/arkplus downloads rather than in +# spleen/'s env — that env exists for the MSD scripts' MONAI/torch, which this needs none of. +download-spleen-flip-format-data: + @uv run --no-project --with huggingface_hub python spleen/download_spleen_flip_format_dataset.py \ + --cache-dir $(HF_CACHE) \ + --output-dir $(DATA_DIR)/spleen \ + --checkpoint-dir $(DATA_DIR)/model_checkpoints \ + --repo-id $(HF_TEST_DATA_REPO) + +# Download the Ark+ chest X-ray TRAIN splits from Hugging Face and normalise them into the +# layout arkplus_fine_tuning/.env.app expects (data/arkplus/site{1,2}/, gitignored). +# Idempotent — safe to re-run. Large (~6.3 GB). Script + README live in arkplus/ (see its +# README) — the same script backs download-arkplus-eval-data below, parameterised by --sites. +download-arkplus-finetuning-data: + @uv run --no-project --with huggingface_hub python arkplus/download_arkplus_dataset.py \ + --cache-dir $(HF_CACHE) \ + --output-dir $(DATA_DIR)/arkplus \ + --repo-id $(ARKPLUS_DATA_REPO) \ + --sites site1 site2 \ + --label training \ + --next-step "make -C fl-tutorials run-tutorial TUTORIAL=arkplus_fine_tuning" \ + --write-marker + +# Download the Ark+ chest X-ray HOLD-OUT splits from Hugging Face and normalise them into the +# layout the two Ark+ evaluation tutorials' .env.app expect (data/arkplus/site{1,2}_holdoff/, +# gitignored). Idempotent — safe to re-run (~1.6 GB). +download-arkplus-eval-data: + @uv run --no-project --with huggingface_hub python arkplus/download_arkplus_dataset.py \ + --cache-dir $(HF_CACHE) \ + --output-dir $(DATA_DIR)/arkplus \ + --repo-id $(ARKPLUS_DATA_REPO) \ + --sites site1_holdoff site2_holdoff \ + --label hold-out \ + --next-step "make -C fl-tutorials run-tutorial TUTORIAL=arkplus_baseline_classification_evaluation (or arkplus_multimodel_classification_evaluation)" + +# Data enrichment: upload the MSD spleen labels into a real FLIP project's XNAT, so the spleen +# apps find a label_*.nii.gz beside each pulled input_*.nii.gz. Needs NUM_CASES=41 on the MSD +# download above to cover every mapped accession. +# +# Enrich EVERY Trust in the project, not one: each Trust's XNAT holds only its own studies, so a +# Trust left without labels fails training at the zero-pairs guard. One invocation covers the +# roster — pass a space-separated XNAT_URLS (credentials from XNAT_USER/XNAT_PASS), or repeat +# XNAT_CREDENTIALS_FILES for per-Trust logins. With neither, it falls back to a single server from +# XNAT_HOST/XNAT_USER/XNAT_PASS. TRUST=N filters the manifest by the OMOP source_trust column and +# is rarely needed: the whole mapping goes to every server and each ignores the others' studies. +# +# The script is single-copy and backend-agnostic — enrichment happens once per FLIP project, in +# XNAT, and both backends read the result. Only the label source differs per variant, so +# SPLEEN_LABELS_DIR defaults by FL_BACKEND (MSD build vs FLIP-format tree) and can be overridden. +# It runs via `uv run --no-project --with ` rather than spleen/'s env, so uploading +# labels never drags in the simulator's MONAI/torch stack. +SPLEEN_LABELS_DIR ?= $(abspath $(if $(filter flower,$(FL_BACKEND)),$(DATA_DIR)/spleen/accession-resources,$(DATA_DIR)/spleen/images)) +# upload-spleen-labels runs against the in-tree flip-utils, not the PyPI release, so the enrichment +# CLI always matches this checkout (flip.xnat may be newer here than the last published version). +FLIP_UTILS_DIR ?= $(abspath ../../flip-utils) + +upload-spleen-labels: + @[ -n "$(FLIP_PROJECT_ID)" ] || (echo "❌ FLIP_PROJECT_ID is required. Usage: make upload-spleen-labels FLIP_PROJECT_ID= [XNAT_URLS=\"http://host1 http://host2\"] [DRY_RUN=1]"; exit 1) + @uv run --no-project --with $(FLIP_UTILS_DIR) python spleen/upload_spleen_labels_to_xnat.py \ + --flip-project-id $(FLIP_PROJECT_ID) \ + --labels-dir $(SPLEEN_LABELS_DIR) \ + $(if $(TRUST),--trust $(TRUST)) \ + $(foreach url,$(XNAT_URLS),--xnat-url $(url)) \ + $(foreach cfg,$(XNAT_CREDENTIALS_FILES) $(XNAT_CREDENTIALS_FILE),--credentials-file $(cfg)) \ + $(if $(OVERWRITE),--overwrite) \ + $(if $(ALLOW_NO_OP),--allow-no-op) \ + $(if $(REQUIRE_FULL_COVERAGE),--require-full-coverage) \ + $(if $(DRY_RUN),--dry-run) diff --git a/fl-tutorials/datasets/README.md b/fl-tutorials/datasets/README.md new file mode 100644 index 000000000..4d0f0174b --- /dev/null +++ b/fl-tutorials/datasets/README.md @@ -0,0 +1,80 @@ + + +# Tutorial datasets + +Shared download/derive/enrich tooling for the FL tutorial datasets. Datasets are +**backend-agnostic** — a tutorial's NVFLARE and Flower twins train on the same data — and +**many-to-one** — one dataset serves several tutorials (spleen alone backs segmentation, +evaluation and diffusion) — so their tooling lives here once instead of duplicated per +backend tree. Everything downloads at run time; nothing is committed. + +All outputs land under the shared, gitignored [`fl-tutorials/data/`](../) root, so one +download serves both backends' harnesses (the NVFLARE simulator via each tutorial's +`.env.app`, the Flower compose stack via `flower/run-tutorial.sh`). + +Invoke the targets through the fl-tutorials root Makefile (which forwards here): + +```bash +make -C fl-tutorials download-xray-data +make -C fl-tutorials download-spleen-data # MSD build (NUM_CASES=<1-41>, default 10) +make -C fl-tutorials download-spleen-data FL_BACKEND=flower # pre-built FLIP-format tree +make -C fl-tutorials download-arkplus-finetuning-data # large (~6.3 GB) +make -C fl-tutorials download-arkplus-eval-data # (~1.6 GB) +make -C fl-tutorials upload-spleen-labels FLIP_PROJECT_ID= # data enrichment +``` + +| Dataset | Source | Output under `fl-tutorials/data/` | Consumed by | +| --- | --- | --- | --- | +| xray | HF `aicentreflip/flip-fl-base-test-data` | `xrays_mini_300/{accession-resources/, dataframe.csv}` | xray_classification (both backends) | +| spleen (MSD build) | MSD Task09_Spleen | `spleen/{images/, dataframe.csv}` | 3d_spleen_segmentation + evaluation + latent_diffusion_model (NVFLARE sim); enrichment labels | +| spleen (FLIP-format) | HF `aicentreflip/flip-fl-base-test-data` | `spleen/{accession-resources/, sample_get_dataframe_response.csv}` + `model_checkpoints/model.pt` | 3d_spleen_segmentation + evaluation (Flower stack) | +| arkplus | HF `aicentreflip/tutorials-arkplus-cxr-classification` | `arkplus/site{1,2}[,_holdoff]/` | the three Ark+ tutorials (NVFLARE) | + +The two spleen variants coexist in `data/spleen/` — the FLIP-format download removes only +its own outputs, never an MSD build beside it. + +## Per-dataset scripts + +[`spleen/`](spleen/) owns the spleen scripts and their uv project (`pyproject.toml` — MONAI, +pandas, natsort; `uv.lock` is gitignored): + +- `download_spleen_dataset.py` — fetch MSD spleen cases and reorganise each subject to hold + image + label. +- `create_spleen_accession_csv.py` — build the `accession_id` dataframe the trainers read in + LOCAL_DEV. +- `upload_spleen_labels_to_xnat.py` — the data-enrichment step: push `label_*.nii.gz` files + into a real FLIP project's XNAT (see the + [spleen tutorial README](../nvflare/image_segmentation/3d_spleen_segmentation/README.md) + for the full walkthrough, and the repo-root `CLAUDE.md` for its `e2e_smoke` wiring). Runs + against the in-tree `flip-utils`, not `spleen/`'s env. +- `download_spleen_flip_format_dataset.py` — fetch the pre-built FLIP-format tree (fixed + 6-case snapshot) plus the evaluation checkpoint from Hugging Face, replacing only its own + outputs in `data/spleen/`. A pure Hugging Face fetch, so like the xray/arkplus scripts it + runs via `uv run --no-project --with huggingface_hub`, not in `spleen/`'s env. + +[`xrays_mini_300/`](xrays_mini_300/) owns the single x-ray script — no dedicated uv project, +it runs via `uv run --no-project --with huggingface_hub`, the same way `upload-spleen-labels` +runs against `flip-utils` without adopting `spleen/`'s env: + +- `download_xrays_dataset.py` — fetch the Hugging Face snapshot and normalise it into + `accession-resources/` + `dataframe.csv`. + +[`arkplus/`](arkplus/) owns the single arkplus script — no dedicated uv project, it runs via +`uv run --no-project --with huggingface_hub`, the same way `upload-spleen-labels` runs +against `flip-utils` without adopting `spleen/`'s env: + +- `download_arkplus_dataset.py` — fetch the given site folders (TRAIN or HOLD-OUT) from + Hugging Face and normalise each into `accession-resources/` + + `sample_get_dataframe_response.csv`. Parameterised by `--sites`, so one script backs both + `download-arkplus-finetuning-data` and `download-arkplus-eval-data`. diff --git a/fl-tutorials/datasets/arkplus/README.md b/fl-tutorials/datasets/arkplus/README.md new file mode 100644 index 000000000..655402739 --- /dev/null +++ b/fl-tutorials/datasets/arkplus/README.md @@ -0,0 +1,36 @@ + + +# arkplus + +Chest X-ray splits backing the three Ark+ tutorials (NVFLARE), sourced from the HF dataset +`aicentreflip/tutorials-arkplus-cxr-classification`: the TRAIN splits (`site1`, `site2`) back +`arkplus_fine_tuning`; the HOLD-OUT splits (`site1_holdoff`, `site2_holdoff`) back the two +evaluation tutorials. + +No dedicated uv project — `download_arkplus_dataset.py`'s only dependency is +`huggingface_hub`, so it runs via `uv run --no-project --with huggingface_hub`, the same way +`upload-spleen-labels` runs against `flip-utils` without adopting `spleen/`'s env. + +- `download_arkplus_dataset.py` — fetch the given site folders from Hugging Face and + normalise each into `/{accession-resources/, sample_get_dataframe_response.csv}`. + One script, parameterised by `--sites`, backs both Makefile targets below; only the TRAIN + call passes `--write-marker` (`arkplus_fine_tuning/Makefile`'s `reproduce-overhead` checks + that marker — not just directory existence — before skipping a re-download). + +Invoke via the fl-tutorials root Makefile (see [`../README.md`](../README.md)): + +```bash +make -C fl-tutorials download-arkplus-finetuning-data # large (~6.3 GB) +make -C fl-tutorials download-arkplus-eval-data # (~1.6 GB) +``` diff --git a/fl-tutorials/datasets/arkplus/download_arkplus_dataset.py b/fl-tutorials/datasets/arkplus/download_arkplus_dataset.py new file mode 100644 index 000000000..57e98996d --- /dev/null +++ b/fl-tutorials/datasets/arkplus/download_arkplus_dataset.py @@ -0,0 +1,95 @@ +# Copyright (c) 2026 Guy's and St Thomas' NHS Foundation Trust & King's College London +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os +import shutil + +from huggingface_hub import snapshot_download + + +def download_arkplus_splits(repo_id, cache_dir, output_dir, sites, write_marker): + """ + Downloads the given Ark+ chest X-ray splits from Hugging Face and normalises each into the layout the + tutorials' .env.app files expect (//{accession-resources/, sample_get_dataframe_response.csv}). + + Args: + repo_id (str): Hugging Face dataset repo to pull from. + cache_dir (str): Directory the raw Hugging Face snapshot is fetched into. + output_dir (str): Parent directory the per-site output folders are written under. + sites (list[str]): Site folder names to fetch, e.g. ["site1", "site2"]. + write_marker (bool): Write a `.download-complete` marker into each site folder once every site has + copied successfully. `arkplus_fine_tuning/Makefile`'s `reproduce-overhead` reads this marker (not + just directory existence) so an interrupted download/copy can't be mistaken for complete data. + """ + snapshot_download( + repo_id=repo_id, + repo_type="dataset", + local_dir=cache_dir, + allow_patterns=[f"{site}/*" for site in sites], + ) + + for site in sites: + site_output = os.path.join(output_dir, site) + if os.path.exists(site_output): + shutil.rmtree(site_output) + os.makedirs(site_output) + + src_dir = os.path.join(cache_dir, site) + shutil.copytree(os.path.join(src_dir, "accession-resources"), os.path.join(site_output, "accession-resources")) + shutil.copy( + os.path.join(src_dir, "sample_get_dataframe_response.csv"), + os.path.join(site_output, "sample_get_dataframe_response.csv"), + ) + + # Written last, only once every site above copied without error — see the write_marker docstring note. + if write_marker: + for site in sites: + open(os.path.join(output_dir, site, ".download-complete"), "w").close() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--cache-dir", required=True, help="directory to fetch the raw Hugging Face snapshot into." + ) + parser.add_argument( + "--output-dir", required=True, help="parent directory for the per-site output folders (gitignored)." + ) + parser.add_argument("--repo-id", required=True, help="Hugging Face dataset repo id.") + parser.add_argument("--sites", required=True, nargs="+", help="site folder names to fetch, e.g. site1 site2.") + parser.add_argument( + "--label", default="", help="human-readable split name for the progress messages, e.g. 'training'." + ) + parser.add_argument( + "--next-step", default="", help="hint printed on success, e.g. the run-tutorial command to try next." + ) + parser.add_argument( + "--write-marker", + action="store_true", + help="write a .download-complete marker into each site folder once every site has copied.", + ) + args = parser.parse_args() + + label = f"{args.label} " if args.label else "" + print(f"⬇️ Downloading Ark+ {label}splits from Hugging Face ({args.repo_id})...") + download_arkplus_splits( + repo_id=args.repo_id, + cache_dir=args.cache_dir, + output_dir=args.output_dir, + sites=args.sites, + write_marker=args.write_marker, + ) + sites_str = "{" + ",".join(args.sites) + "}" + message = f"✅ Ark+ {label}data ready at {os.path.abspath(args.output_dir)}/{sites_str}/" + if args.next_step: + message += f" — {args.next_step}" + print(message) diff --git a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/create_spleen_accession_csv.py b/fl-tutorials/datasets/spleen/create_spleen_accession_csv.py similarity index 100% rename from fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/create_spleen_accession_csv.py rename to fl-tutorials/datasets/spleen/create_spleen_accession_csv.py diff --git a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/download_spleen_dataset.py b/fl-tutorials/datasets/spleen/download_spleen_dataset.py similarity index 100% rename from fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/download_spleen_dataset.py rename to fl-tutorials/datasets/spleen/download_spleen_dataset.py diff --git a/fl-tutorials/datasets/spleen/download_spleen_flip_format_dataset.py b/fl-tutorials/datasets/spleen/download_spleen_flip_format_dataset.py new file mode 100644 index 000000000..d898cb248 --- /dev/null +++ b/fl-tutorials/datasets/spleen/download_spleen_flip_format_dataset.py @@ -0,0 +1,92 @@ +# Copyright (c) 2026 Guy's and St Thomas' NHS Foundation Trust & King's College London +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os +import shutil + +from huggingface_hub import snapshot_download + +REPO_ID = "aicentreflip/flip-fl-base-test-data" +# The repo's internal layout nests the dataset under a folder matching the repo's own name. +REPO_SUBDIR = "flip-fl-base-test-data/spleen" +REPO_CHECKPOINT_SUBDIR = "flip-fl-base-test-data/checkpoints" +# The two outputs this script owns under output_dir. Only these are removed before a re-download: +# the MSD build (images/, dataframe.csv) shares data/spleen/ and must survive a re-run of this variant. +OUTPUTS = ("accession-resources", "sample_get_dataframe_response.csv") + + +def download_spleen_flip_format_dataset(cache_dir, output_dir, checkpoint_dir, repo_id=REPO_ID): + """ + Downloads the pre-built FLIP-format spleen tree (fixed 6-case snapshot) and the evaluation-tutorial checkpoint + from Hugging Face and normalises them into the layout the Flower compose stack mounts. + + Args: + cache_dir (str): Directory the raw Hugging Face snapshot is fetched into. + output_dir (str): The shared spleen data directory. Only this variant's own outputs (accession-resources/, + sample_get_dataframe_response.csv) are removed and recreated, so an MSD build beside them survives. + checkpoint_dir (str): Directory the evaluation checkpoint is copied into as model.pt. + repo_id (str): Hugging Face dataset repo to pull from. + """ + snapshot_download( + repo_id=repo_id, + repo_type="dataset", + local_dir=cache_dir, + allow_patterns=[f"{REPO_SUBDIR}/*", f"{REPO_CHECKPOINT_SUBDIR}/*"], + ) + + src_dir = os.path.join(cache_dir, REPO_SUBDIR) + + for output in OUTPUTS: + target = os.path.join(output_dir, output) + if os.path.isdir(target): + shutil.rmtree(target) + elif os.path.exists(target): + os.remove(target) + os.makedirs(output_dir, exist_ok=True) + + shutil.copytree(os.path.join(src_dir, "accession-resources"), os.path.join(output_dir, "accession-resources")) + shutil.copy( + os.path.join(src_dir, "sample_get_dataframe_response.csv"), + os.path.join(output_dir, "sample_get_dataframe_response.csv"), + ) + + os.makedirs(checkpoint_dir, exist_ok=True) + shutil.copy(os.path.join(cache_dir, REPO_CHECKPOINT_SUBDIR, "model.pt"), os.path.join(checkpoint_dir, "model.pt")) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--cache-dir", required=True, help="directory to fetch the raw Hugging Face snapshot into." + ) + parser.add_argument( + "--output-dir", + required=True, + help="the shared spleen data directory (gitignored); only this variant's own outputs are replaced.", + ) + parser.add_argument( + "--checkpoint-dir", required=True, help="directory the evaluation checkpoint is copied into as model.pt." + ) + parser.add_argument("--repo-id", default=REPO_ID, help="Hugging Face dataset repo id.") + args = parser.parse_args() + + print(f"⬇️ Downloading FLIP-format spleen dataset from Hugging Face ({args.repo_id})...") + download_spleen_flip_format_dataset( + cache_dir=args.cache_dir, + output_dir=args.output_dir, + checkpoint_dir=args.checkpoint_dir, + repo_id=args.repo_id, + ) + print( + f"✅ FLIP-format spleen data ready at {os.path.abspath(args.output_dir)}/ (+ checkpoint) — " + "make -C fl-tutorials run-tutorial TUTORIAL=3d_spleen_segmentation FL_BACKEND=flower" + ) diff --git a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/pyproject.toml b/fl-tutorials/datasets/spleen/pyproject.toml similarity index 63% rename from fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/pyproject.toml rename to fl-tutorials/datasets/spleen/pyproject.toml index 7fd2d3992..40df12849 100644 --- a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/pyproject.toml +++ b/fl-tutorials/datasets/spleen/pyproject.toml @@ -1,7 +1,7 @@ [project] -name = "spleen-segmentation-tutorial" +name = "spleen-dataset-tools" version = "0.1.0" -description = "Local uv environment for the 3D spleen segmentation tutorial utilities." +description = "Local uv environment for the shared spleen dataset tooling (MSD download + accession CSV)." requires-python = ">=3.12,<3.14" dependencies = [ "monai>=1.5.1", diff --git a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/upload_spleen_labels_to_xnat.py b/fl-tutorials/datasets/spleen/upload_spleen_labels_to_xnat.py similarity index 96% rename from fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/upload_spleen_labels_to_xnat.py rename to fl-tutorials/datasets/spleen/upload_spleen_labels_to_xnat.py index 09ae2280e..9757ca659 100644 --- a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/upload_spleen_labels_to_xnat.py +++ b/fl-tutorials/datasets/spleen/upload_spleen_labels_to_xnat.py @@ -28,8 +28,8 @@ derived from the converted image, so running early skips every scan. This is backend-agnostic: enrichment happens once per FLIP project, in XNAT, and both the NVFLARE -and Flower spleen tutorials read the result. Point ``--labels-dir`` at whichever backend's spleen -download you have. +and Flower spleen tutorials read the result. Point ``--labels-dir`` at whichever spleen download +variant you have (see the datasets README). """ import argparse @@ -180,14 +180,14 @@ def fetch_accession_map(url: str | None = None, cache_dir: Path | None = None) - INCOMPLETE_DOWNLOAD_HELP = """ The two spleen downloads differ, so check the one --labels-dir points at: - NVFLARE (data/spleen/images) make -C fl-tutorials/nvflare download-spleen-data NUM_CASES=41 - Flower (data/spleen/accession-resources) the HF snapshot ships a fixed 6-case subset, and - ignores NUM_CASES — point --labels-dir at the - NVFLARE download for full coverage.""" -"""Remediation text naming both backends. + MSD build (data/spleen/images) make -C fl-tutorials download-spleen-data NUM_CASES=41 + FLIP-format (data/spleen/accession-resources) the HF snapshot ships a fixed 6-case subset, and + ignores NUM_CASES — point --labels-dir at the + MSD download for full coverage.""" +"""Remediation text naming both download variants. Printed unconditionally rather than inferred from ``--labels-dir``: guessing wrong sends the reader -to a command that cannot help, and ``NUM_CASES`` genuinely does nothing on the Flower path. +to a command that cannot help, and ``NUM_CASES`` genuinely does nothing on the FLIP-format path. """ diff --git a/fl-tutorials/datasets/xrays_mini_300/README.md b/fl-tutorials/datasets/xrays_mini_300/README.md new file mode 100644 index 000000000..bd5b03aee --- /dev/null +++ b/fl-tutorials/datasets/xrays_mini_300/README.md @@ -0,0 +1,30 @@ + + +# xrays_mini_300 + +Reference x-ray dataset backing the `xray_classification` tutorial (both backends). Sourced +from the `xrays_mini_300/` subtree of the HF dataset `aicentreflip/flip-fl-base-test-data`. + +No dedicated uv project — `download_xrays_dataset.py`'s only dependency is `huggingface_hub`, +so it runs via `uv run --no-project --with huggingface_hub`, the same way +`upload-spleen-labels` runs against `flip-utils` without adopting `spleen/`'s env. + +- `download_xrays_dataset.py` — fetch the Hugging Face snapshot and normalise it into + `accession-resources/` + `dataframe.csv` under the shared `fl-tutorials/data/xrays_mini_300/`. + +Invoke via the fl-tutorials root Makefile (see [`../README.md`](../README.md)): + +```bash +make -C fl-tutorials download-xray-data +``` diff --git a/fl-tutorials/datasets/xrays_mini_300/download_xrays_dataset.py b/fl-tutorials/datasets/xrays_mini_300/download_xrays_dataset.py new file mode 100644 index 000000000..66f0d861e --- /dev/null +++ b/fl-tutorials/datasets/xrays_mini_300/download_xrays_dataset.py @@ -0,0 +1,68 @@ +# Copyright (c) 2026 Guy's and St Thomas' NHS Foundation Trust & King's College London +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os +import shutil + +from huggingface_hub import snapshot_download + +REPO_ID = "aicentreflip/flip-fl-base-test-data" +# The repo's internal layout nests the dataset under a folder matching the repo's own name. +REPO_SUBDIR = "flip-fl-base-test-data/xrays_mini_300" + + +def download_xrays_dataset(cache_dir, output_dir, repo_id=REPO_ID): + """ + Downloads the x-ray reference dataset from Hugging Face and normalises it into the layout the FL tutorial + harnesses read (accession-resources/, dataframe.csv). + + Args: + cache_dir (str): Directory the raw Hugging Face snapshot is fetched into. + output_dir (str): Target directory for the normalised dataset. Removed and recreated on each run. + repo_id (str): Hugging Face dataset repo to pull from. + """ + snapshot_download( + repo_id=repo_id, + repo_type="dataset", + local_dir=cache_dir, + allow_patterns=[f"{REPO_SUBDIR}/*"], + ) + + src_dir = os.path.join(cache_dir, REPO_SUBDIR) + + if os.path.exists(output_dir): + shutil.rmtree(output_dir) + os.makedirs(output_dir) + + shutil.copytree(os.path.join(src_dir, "accession-resources"), os.path.join(output_dir, "accession-resources")) + shutil.copy( + os.path.join(src_dir, "sample_get_dataframe_response.csv"), os.path.join(output_dir, "dataframe.csv") + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--cache-dir", required=True, help="directory to fetch the raw Hugging Face snapshot into." + ) + parser.add_argument( + "--output-dir", required=True, help="target directory for the normalised dataset (gitignored)." + ) + parser.add_argument("--repo-id", default=REPO_ID, help="Hugging Face dataset repo id.") + args = parser.parse_args() + + print(f"⬇️ Downloading x-ray dataset from Hugging Face ({args.repo_id})...") + download_xrays_dataset(cache_dir=args.cache_dir, output_dir=args.output_dir, repo_id=args.repo_id) + print( + f"✅ x-ray data ready at {os.path.abspath(args.output_dir)}/ — " + "make -C fl-tutorials run-tutorial TUTORIAL=xray_classification" + ) diff --git a/fl-tutorials/flower/3d_spleen_segmentation/README.md b/fl-tutorials/flower/3d_spleen_segmentation/README.md index aadb684b7..688f07ae2 100644 --- a/fl-tutorials/flower/3d_spleen_segmentation/README.md +++ b/fl-tutorials/flower/3d_spleen_segmentation/README.md @@ -11,13 +11,6 @@ limitations under the License. --> ---- - -tags: [quickstart] -dataset: [spleen] -framework: [monai] ---- - # Federated Learning with MONAI and Flower (Quickstart Example) This example of Flower uses a small MONAI UNet based on FLIP's implementation and a training-only `ClientApp`. It reads NIfTI data from the local `./data` folder and does not write any outputs. @@ -213,17 +206,19 @@ matters because each Trust's XNAT holds only its own studies and a Trust left wi > **This tutorial's download covers only part of the cohort.** `download-spleen-data FL_BACKEND=flower` > pulls a fixed 6-case HF snapshot and ignores `NUM_CASES`, while the accession mapping spans 41. Enriching > from it succeeds but leaves most of the cohort unlabelled, and the command says so. For full coverage use -> the NVFLARE download (`make -C fl-tutorials/nvflare download-spleen-data NUM_CASES=41`) and point +> the MSD download (`make -C fl-tutorials download-spleen-data NUM_CASES=41`) and point > `SPLEEN_LABELS_DIR` at it — the labels are backend-agnostic once they are in XNAT. Enrichment is **backend-agnostic**: the labels live in XNAT, so a project enriched once can be trained by -either backend. This target deliberately delegates to the single copy of the upload script in -`fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/utils/`, passing this tutorial's own data -directory — see that tutorial's README for the full walkthrough and options. +either backend. The upload script is a single copy in +[`fl-tutorials/datasets/spleen/`](../../datasets/spleen); with `FL_BACKEND=flower` the target points it at +this tutorial's FLIP-format tree — see the +[NVFLARE spleen tutorial's README](../../nvflare/image_segmentation/3d_spleen_segmentation/README.md) for +the full walkthrough and options. ## Data Location -By default, the app reads from: +By default, the app reads from the shared gitignored `fl-tutorials/data/` root: -- `data/sample_get_dataframe_response.csv` -- `data/accession-resources` +- `data/spleen/sample_get_dataframe_response.csv` +- `data/spleen/accession-resources` diff --git a/fl-tutorials/flower/3d_spleen_segmentation_evaluation/README.md b/fl-tutorials/flower/3d_spleen_segmentation_evaluation/README.md index 9756a36b4..5c09b2fff 100644 --- a/fl-tutorials/flower/3d_spleen_segmentation_evaluation/README.md +++ b/fl-tutorials/flower/3d_spleen_segmentation_evaluation/README.md @@ -11,13 +11,6 @@ limitations under the License. --> ---- - -tags: [quickstart] -dataset: [spleen] -framework: [monai] ---- - # Federated Evaluation with MONAI and Flower This example uses a MONAI UNet for 3D spleen segmentation in an evaluation-only mode. It loads a pre-trained model checkpoint and performs federated evaluation across multiple client nodes. diff --git a/fl-tutorials/flower/Makefile b/fl-tutorials/flower/Makefile index b9fca1ce0..a6e3cc787 100644 --- a/fl-tutorials/flower/Makefile +++ b/fl-tutorials/flower/Makefile @@ -15,65 +15,31 @@ # bind-mounted (LOCAL_DEV), submits the job to the fl-api control plane, waits, and # tears down. The root `make -C fl-tutorials … FL_BACKEND=flower` forwards here. # -# make download-xray-data # FLIP-format x-ray data from HF -# make download-spleen-data # FLIP-format MSD-spleen data from HF -# make upload-spleen-labels FLIP_PROJECT_ID= # data enrichment: labels into a Trust's XNAT # make list-tutorials -# make run-tutorial TUTORIAL=xray_classification # run 'make download-xray-data' first -# make run-tutorial TUTORIAL=3d_spleen_segmentation # needs download-spleen-data +# make run-tutorial TUTORIAL=xray_classification +# make run-tutorial TUTORIAL=3d_spleen_segmentation # make run-all-tutorials # -# Datasets land (gitignored) under data/ in FLIP accession-resources layout, the -# same Hugging Face source the NVFLARE tutorials use. +# Every tutorial needs its dataset downloaded first — +# `make -C fl-tutorials download--data FL_BACKEND=flower`. The dataset tooling is +# shared with the NVFLARE harness in fl-tutorials/datasets/ and lands under the shared +# gitignored fl-tutorials/data/, which run-tutorial.sh bind-mounts into the stack. REPO_ROOT := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/../..) -HF_DATA_REPO := aicentreflip/flip-fl-base-test-data # Tutorials = immediate subdirs carrying an app/ (xray_classification, # 3d_spleen_segmentation, 3d_spleen_segmentation_evaluation). TUTORIAL_DIRS := $(sort $(patsubst %/app/,%,$(dir $(wildcard */app/.)))) -.PHONY: list-tutorials run-tutorial run-all-tutorials download-xray-data download-spleen-data upload-spleen-labels - -# Data enrichment: upload the MSD spleen labels into a real FLIP project's XNAT. -# -# Deliberately delegates to the NVFLARE tree rather than duplicating the script. Enrichment is a -# one-time, backend-agnostic project step: the labels land in XNAT, and both backends then read -# them through flip.get_by_accession_number. Only the local label directory differs — Flower pulls -# a pre-built FLIP-format tree from HF, NVFLARE builds one from MSD — and both use the same -# subject_N/scans/{input,label}_spleen_N.nii.gz layout, so pointing SPLEEN_LABELS_DIR at ours is -# the whole difference. -upload-spleen-labels: - @$(MAKE) --no-print-directory -C $(REPO_ROOT)/fl-tutorials/nvflare upload-spleen-labels \ - SPLEEN_LABELS_DIR=$(abspath data/spleen/accession-resources) - -# Download the FLIP-format x-ray dataset from Hugging Face into data/xrays_mini_300/ -# (gitignored). Idempotent. Mirrors fl-tutorials/nvflare's download-xray-data. -download-xray-data: - @echo "⬇️ Downloading x-ray dataset from Hugging Face ($(HF_DATA_REPO))..." - @uv run --no-project --with huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$(HF_DATA_REPO)', repo_type='dataset', local_dir='data/.hf', allow_patterns=['flip-fl-base-test-data/xrays_mini_300/*'])" - @rm -rf data/xrays_mini_300 && mkdir -p data/xrays_mini_300 - @cp -r data/.hf/flip-fl-base-test-data/xrays_mini_300/accession-resources data/xrays_mini_300/accession-resources - @cp data/.hf/flip-fl-base-test-data/xrays_mini_300/sample_get_dataframe_response.csv data/xrays_mini_300/sample_get_dataframe_response.csv - @echo "✅ x-ray data ready at data/xrays_mini_300/ — make run-tutorial TUTORIAL=xray_classification" - -# Download the FLIP-format spleen dataset (accession-resources + dataframe) from HF -# into data/spleen/ (gitignored), shared by the segmentation + evaluation tutorials. -download-spleen-data: - @echo "⬇️ Downloading spleen dataset from Hugging Face ($(HF_DATA_REPO))..." - @uv run --no-project --with huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$(HF_DATA_REPO)', repo_type='dataset', local_dir='data/.hf', allow_patterns=['flip-fl-base-test-data/spleen/*', 'flip-fl-base-test-data/checkpoints/*'])" - @rm -rf data/spleen && mkdir -p data/spleen - @cp -r data/.hf/flip-fl-base-test-data/spleen/accession-resources data/spleen/accession-resources - @cp data/.hf/flip-fl-base-test-data/spleen/sample_get_dataframe_response.csv data/spleen/sample_get_dataframe_response.csv - @mkdir -p data/model_checkpoints && cp data/.hf/flip-fl-base-test-data/checkpoints/model.pt data/model_checkpoints/model.pt - @echo "✅ spleen data ready at data/spleen/ (+ checkpoint) — make run-tutorial TUTORIAL=3d_spleen_segmentation" +.PHONY: list-tutorials run-tutorial run-all-tutorials list-tutorials: @echo "Available Flower tutorials (pass the name to 'make run-tutorial TUTORIAL=…'):" @for d in $(TUTORIAL_DIRS); do echo " - $$d"; done # Run one tutorial on the standalone Flower stack (up → submit → wait → down). -# Every tutorial needs its dataset: run 'make download--data' first. +# Every tutorial needs its dataset first: +# `make -C fl-tutorials download--data FL_BACKEND=flower`. run-tutorial: @bash $(REPO_ROOT)/fl-tutorials/flower/run-tutorial.sh "$(TUTORIAL)" diff --git a/fl-tutorials/flower/run-tutorial.sh b/fl-tutorials/flower/run-tutorial.sh index 2efcf483c..5ec01d816 100755 --- a/fl-tutorials/flower/run-tutorial.sh +++ b/fl-tutorials/flower/run-tutorial.sh @@ -27,20 +27,29 @@ list() { for d in "$HERE"/*/app; do basename "$(dirname "$d")"; done; } if [ -z "$TUTORIAL" ]; then echo "Set TUTORIAL=. Available:"; list | sed 's/^/ - /'; exit 1; fi if [ ! -d "$HERE/$TUTORIAL/app" ]; then echo "❌ Unknown tutorial '$TUTORIAL'. Available:"; list | sed 's/^/ - /'; exit 1; fi -# Per-tutorial dev data (LOCAL_DEV reads DEV_IMAGES_DIR / DEV_DATAFRAME). Spleen and xray -# both need their HF dataset downloaded first. +# Per-tutorial dev data (LOCAL_DEV reads DEV_IMAGES_DIR / DEV_DATAFRAME), served from the +# shared gitignored fl-tutorials/data/ root (see fl-tutorials/datasets/). Every tutorial +# needs its dataset downloaded first. +DATA_ROOT="$(cd "$HERE/.." && pwd)/data" case "$TUTORIAL" in 3d_spleen_segmentation|3d_spleen_segmentation_evaluation) - export DEV_IMAGES_DIR="$HERE/data/spleen/accession-resources" - export DEV_DATAFRAME="$HERE/data/spleen/sample_get_dataframe_response.csv" ;; + export DEV_IMAGES_DIR="$DATA_ROOT/spleen/accession-resources" + export DEV_DATAFRAME="$DATA_ROOT/spleen/sample_get_dataframe_response.csv" + DATASET_TARGET=spleen ;; xray_classification) - export DEV_IMAGES_DIR="$HERE/data/xrays_mini_300/accession-resources" - export DEV_DATAFRAME="$HERE/data/xrays_mini_300/sample_get_dataframe_response.csv" ;; + export DEV_IMAGES_DIR="$DATA_ROOT/xrays_mini_300/accession-resources" + export DEV_DATAFRAME="$DATA_ROOT/xrays_mini_300/dataframe.csv" + DATASET_TARGET=xray ;; *) echo "❌ No data mapping for '$TUTORIAL'"; exit 1 ;; esac if [ -n "${DEV_IMAGES_DIR:-}" ] && [ ! -d "$DEV_IMAGES_DIR" ]; then echo "❌ Dataset missing: $DEV_IMAGES_DIR" - echo " Run: make -C $REPO_ROOT/fl-tutorials download-$( [ "${TUTORIAL#xray}" != "$TUTORIAL" ] && echo xray || echo spleen )-data FL_BACKEND=flower" + echo " Run: make -C $REPO_ROOT/fl-tutorials download-$DATASET_TARGET-data FL_BACKEND=flower" + exit 1 +fi +if [ -n "${DEV_DATAFRAME:-}" ] && [ ! -f "$DEV_DATAFRAME" ]; then + echo "❌ Dataset missing: $DEV_DATAFRAME" + echo " Run: make -C $REPO_ROOT/fl-tutorials download-$DATASET_TARGET-data FL_BACKEND=flower" exit 1 fi diff --git a/fl-tutorials/nvflare/Makefile b/fl-tutorials/nvflare/Makefile index 12da09750..ab15ba200 100644 --- a/fl-tutorials/nvflare/Makefile +++ b/fl-tutorials/nvflare/Makefile @@ -15,123 +15,18 @@ # `full` ML extra. Requires a GPU, and the tutorial's dataset downloaded first (see each # tutorial's README); `make export` inside a tutorial needs neither. # -# make download-xray-data # fetch the x-ray dataset from HF -# make download-spleen-data # fetch + build the MSD spleen dataset -# make upload-spleen-labels FLIP_PROJECT_ID= # data enrichment: push labels into a Trust's XNAT -# make download-arkplus-finetuning-data # fetch the Ark+ train splits from HF -# make download-arkplus-eval-data # fetch the Ark+ hold-out splits from HF # make list-tutorials # make run-tutorial TUTORIAL=xray_classification # make run-all-tutorials # runs every tutorial (heavy; stops on first failure) # -# Datasets land (gitignored) under data/ where each tutorial's .env.app expects them: -# - xray_classification: reference dataset from Hugging Face (download-xray-data) -# - the spleen tutorials (segmentation / evaluation / synthesis) share the MSD -# spleen dataset (download-spleen-data); override case count with NUM_CASES=. -# - the three Ark+ tutorials share aicentreflip/tutorials-arkplus-cxr-classification: -# arkplus_fine_tuning uses download-arkplus-finetuning-data (data/arkplus/site{1,2}/); the two -# Ark+ evaluation tutorials use download-arkplus-eval-data (data/arkplus/site{1,2}_holdoff/). +# Dataset downloads live in fl-tutorials/datasets/ (shared with the Flower harness), and land +# under the shared gitignored fl-tutorials/data/ where each tutorial's .env.app points — +# `make -C fl-tutorials download--data` (see fl-tutorials/datasets/README.md). # Discover tutorials: any // that carries a .env.app. TUTORIAL_DIRS := $(sort $(patsubst %/,%,$(dir $(wildcard */*/.env.app)))) -# Hugging Face dataset backing the xray_classification tutorial. -XRAY_DATA_REPO := aicentreflip/flip-fl-base-test-data -# Hugging Face dataset backing the three Ark+ tutorials (site{1,2} train + hold-out splits). -ARKPLUS_DATA_REPO := aicentreflip/tutorials-arkplus-cxr-classification -# Number of MSD spleen cases to download for the spleen tutorials (1-41). -NUM_CASES ?= 10 -SPLEEN_DIR := image_segmentation/3d_spleen_segmentation -# Where download-spleen-data lands the subject_N/scans/{input,label}_spleen_N.nii.gz trees that -# upload-spleen-labels reads. Flower's Makefile overrides this to its own spleen download. -SPLEEN_LABELS_DIR ?= data/spleen/images -# upload-spleen-labels runs against the in-tree flip-utils, not the PyPI release, so the enrichment -# CLI always matches this checkout (flip.xnat may be newer here than the last published version). -FLIP_UTILS_DIR ?= $(abspath ../../flip-utils) - -.PHONY: list-tutorials run-tutorial run-all-tutorials download-xray-data download-spleen-data download-arkplus-finetuning-data download-arkplus-eval-data upload-spleen-labels - -# Download the x-ray reference dataset from Hugging Face and normalise it into the -# layout image_classification/xray_classification/.env.app expects (data/xrays_mini_300/, -# gitignored). Idempotent — safe to re-run. -download-xray-data: - @echo "⬇️ Downloading x-ray dataset from Hugging Face ($(XRAY_DATA_REPO))..." - @uv run --no-project --with huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$(XRAY_DATA_REPO)', repo_type='dataset', local_dir='data/.hf', allow_patterns=['flip-fl-base-test-data/xrays_mini_300/*'])" - @rm -rf data/xrays_mini_300 - @mkdir -p data/xrays_mini_300 - @cp -r data/.hf/flip-fl-base-test-data/xrays_mini_300/accession-resources data/xrays_mini_300/accession-resources - @cp data/.hf/flip-fl-base-test-data/xrays_mini_300/sample_get_dataframe_response.csv data/xrays_mini_300/dataframe.csv - @echo "✅ x-ray data ready at data/xrays_mini_300/ — run: make run-tutorial TUTORIAL=xray_classification" - -# Download the Ark+ chest X-ray TRAIN splits from Hugging Face and normalise them into the layout -# image_classification/arkplus_fine_tuning/.env.app expects (data/arkplus/site{1,2}/, gitignored). -# Idempotent — safe to re-run. Large (~6.3 GB). -download-arkplus-finetuning-data: - @echo "⬇️ Downloading Ark+ training splits from Hugging Face ($(ARKPLUS_DATA_REPO))..." - @uv run --no-project --with huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$(ARKPLUS_DATA_REPO)', repo_type='dataset', local_dir='data/.hf', allow_patterns=['site1/*', 'site2/*'])" - @rm -rf data/arkplus/site1 data/arkplus/site2 - @mkdir -p data/arkplus/site1 data/arkplus/site2 - @cp -r data/.hf/site1/accession-resources data/arkplus/site1/accession-resources - @cp data/.hf/site1/sample_get_dataframe_response.csv data/arkplus/site1/sample_get_dataframe_response.csv - @cp -r data/.hf/site2/accession-resources data/arkplus/site2/accession-resources - @cp data/.hf/site2/sample_get_dataframe_response.csv data/arkplus/site2/sample_get_dataframe_response.csv - @# Written last, only on a fully successful run: `experiment`'s dataset-present check in - @# arkplus_fine_tuning/Makefile looks for this marker (not just directory existence), so an - @# interrupted download/copy (e.g. Ctrl+C mid-run) can't be mistaken for complete data on the - @# next `make reproduce-overhead`. - @touch data/arkplus/site1/.download-complete data/arkplus/site2/.download-complete - @echo "✅ Ark+ training data ready at data/arkplus/site{1,2}/ — run: make run-tutorial TUTORIAL=arkplus_fine_tuning" - -# Download the Ark+ chest X-ray HOLD-OUT splits from Hugging Face and normalise them into the layout -# the two Ark+ evaluation tutorials' .env.app expect (data/arkplus/site{1,2}_holdoff/, gitignored). -# Idempotent — safe to re-run (~1.6 GB). -download-arkplus-eval-data: - @echo "⬇️ Downloading Ark+ hold-out splits from Hugging Face ($(ARKPLUS_DATA_REPO))..." - @uv run --no-project --with huggingface_hub python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$(ARKPLUS_DATA_REPO)', repo_type='dataset', local_dir='data/.hf', allow_patterns=['site1_holdoff/*', 'site2_holdoff/*'])" - @rm -rf data/arkplus/site1_holdoff data/arkplus/site2_holdoff - @mkdir -p data/arkplus/site1_holdoff data/arkplus/site2_holdoff - @cp -r data/.hf/site1_holdoff/accession-resources data/arkplus/site1_holdoff/accession-resources - @cp data/.hf/site1_holdoff/sample_get_dataframe_response.csv data/arkplus/site1_holdoff/sample_get_dataframe_response.csv - @cp -r data/.hf/site2_holdoff/accession-resources data/arkplus/site2_holdoff/accession-resources - @cp data/.hf/site2_holdoff/sample_get_dataframe_response.csv data/arkplus/site2_holdoff/sample_get_dataframe_response.csv - @echo "✅ Ark+ hold-out data ready at data/arkplus/site{1,2}_holdoff/ — run: make run-tutorial TUTORIAL=arkplus_baseline_classification_evaluation (or arkplus_multimodel_classification_evaluation)" - -# Download the MSD spleen dataset and build its accession CSV into data/spleen/ (gitignored), -# shared by the segmentation / evaluation / synthesis tutorials. Uses the spleen tutorial's own -# uv project for MONAI. Large download — overridable via NUM_CASES (1-41, default 10). -download-spleen-data: - @echo "⬇️ Downloading MSD spleen dataset ($(NUM_CASES) cases) — large, uses the spleen tutorial's deps..." - @cd $(SPLEEN_DIR) && uv run python utils/download_spleen_dataset.py --output_dir ../../data/spleen/images --num_cases $(NUM_CASES) - @cd $(SPLEEN_DIR) && uv run python utils/create_spleen_accession_csv.py --images_dir ../../data/spleen/images --output_csv ../../data/spleen/dataframe.csv - @echo "✅ spleen data ready at data/spleen/ — run: make run-tutorial TUTORIAL=3d_spleen_segmentation" - -# Data enrichment: upload the MSD spleen labels into a real FLIP project's XNAT, so the spleen -# apps find a label_*.nii.gz beside each pulled input_*.nii.gz. Needs NUM_CASES=41 on the download -# above to cover every mapped accession. -# -# Enrich EVERY Trust in the project, not one: each Trust's XNAT holds only its own studies, so a -# Trust left without labels fails training at the zero-pairs guard. One invocation covers the -# roster — pass a space-separated XNAT_URLS (credentials from XNAT_USER/XNAT_PASS), or repeat -# XNAT_CREDENTIALS_FILES for per-Trust logins. With neither, it falls back to a single server from -# XNAT_HOST/XNAT_USER/XNAT_PASS. TRUST=N filters the manifest by the OMOP source_trust column and -# is rarely needed: the whole mapping goes to every server and each ignores the others' studies. -# -# The script is deliberately single-copy and backend-agnostic — enrichment happens once per FLIP -# project, in XNAT, and both backends read the result. fl-tutorials/flower/Makefile delegates here. -# It runs via `uv run --no-project --with ` rather than the spleen tutorial's env, so -# uploading labels never drags in the simulator's MONAI/torch stack. -upload-spleen-labels: - @[ -n "$(FLIP_PROJECT_ID)" ] || (echo "❌ FLIP_PROJECT_ID is required. Usage: make upload-spleen-labels FLIP_PROJECT_ID= [XNAT_URLS=\"http://host1 http://host2\"] [DRY_RUN=1]"; exit 1) - @uv run --no-project --with $(FLIP_UTILS_DIR) python $(SPLEEN_DIR)/utils/upload_spleen_labels_to_xnat.py \ - --flip-project-id $(FLIP_PROJECT_ID) \ - --labels-dir $(SPLEEN_LABELS_DIR) \ - $(if $(TRUST),--trust $(TRUST)) \ - $(foreach url,$(XNAT_URLS),--xnat-url $(url)) \ - $(foreach cfg,$(XNAT_CREDENTIALS_FILES) $(XNAT_CREDENTIALS_FILE),--credentials-file $(cfg)) \ - $(if $(OVERWRITE),--overwrite) \ - $(if $(ALLOW_NO_OP),--allow-no-op) \ - $(if $(REQUIRE_FULL_COVERAGE),--require-full-coverage) \ - $(if $(DRY_RUN),--dry-run) +.PHONY: list-tutorials run-tutorial run-all-tutorials list-tutorials: @echo "Available NVFLARE tutorials (pass the name to 'make run-tutorial TUTORIAL=…'):" diff --git a/fl-tutorials/nvflare/data b/fl-tutorials/nvflare/data deleted file mode 120000 index 97cd5a9ca..000000000 --- a/fl-tutorials/nvflare/data +++ /dev/null @@ -1 +0,0 @@ -/home/localat24/sandbox/londonaicentre/FLIP/fl-tutorials/nvflare/data \ No newline at end of file diff --git a/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/.env.app b/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/.env.app index 424047650..4a6013b2e 100644 --- a/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/.env.app +++ b/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/.env.app @@ -1,8 +1,8 @@ JOB_TYPE=standard -SITE1_IMAGES_DIR=../../data/arkplus/site1/accession-resources -SITE1_DATAFRAME=../../data/arkplus/site1/sample_get_dataframe_response.csv -SITE2_IMAGES_DIR=../../data/arkplus/site2/accession-resources -SITE2_DATAFRAME=../../data/arkplus/site2/sample_get_dataframe_response.csv +SITE1_IMAGES_DIR=../../../data/arkplus/site1/accession-resources +SITE1_DATAFRAME=../../../data/arkplus/site1/sample_get_dataframe_response.csv +SITE2_IMAGES_DIR=../../../data/arkplus/site2/accession-resources +SITE2_DATAFRAME=../../../data/arkplus/site2/sample_get_dataframe_response.csv RAW_CHECKPOINT=models/Ark6_swinLarge768_ep50.pth.tar FLIP_PROJECT_ID= FLIP_QUERY= diff --git a/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/README.md b/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/README.md index 40ea2bbf1..73ed9acf7 100644 --- a/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/README.md +++ b/fl-tutorials/nvflare/image_classification/arkplus_fine_tuning/README.md @@ -99,7 +99,7 @@ make -C fl-tutorials run-tutorial TUTORIAL=arkplus_fine_tuning `download-arkplus-finetuning-data` pulls the `site1`/`site2` training splits of [`aicentreflip/tutorials-arkplus-cxr-classification`](https://huggingface.co/datasets/aicentreflip/tutorials-arkplus-cxr-classification) -and normalises them into `fl-tutorials/nvflare/data/arkplus/site{1,2}/` (gitignored), matching this +and normalises them into `fl-tutorials/data/arkplus/site{1,2}/` (gitignored), matching this tutorial's `.env.app` defaults (`SITE{1,2}_IMAGES_DIR` / `SITE{1,2}_DATAFRAME`). Requires a GPU + the `flare-fl-base` image to run the simulator. To use your own data instead, point the per-site `.env.app` values at it. diff --git a/fl-tutorials/nvflare/image_classification/xray_classification/.env.app b/fl-tutorials/nvflare/image_classification/xray_classification/.env.app index c521ed1fb..fff3f17c8 100644 --- a/fl-tutorials/nvflare/image_classification/xray_classification/.env.app +++ b/fl-tutorials/nvflare/image_classification/xray_classification/.env.app @@ -1,6 +1,6 @@ JOB_TYPE=standard -DEV_IMAGES_DIR=../../data/xrays_mini_300/accession-resources/ -DEV_DATAFRAME=../../data/xrays_mini_300/dataframe.csv +DEV_IMAGES_DIR=../../../data/xrays_mini_300/accession-resources/ +DEV_DATAFRAME=../../../data/xrays_mini_300/dataframe.csv FLIP_PROJECT_ID= FLIP_QUERY= diff --git a/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/.env.app b/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/.env.app index f02b74c9f..08614d22a 100644 --- a/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/.env.app +++ b/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/.env.app @@ -1,6 +1,6 @@ JOB_TYPE=evaluation -DEV_IMAGES_DIR=../../data/spleen/images -DEV_DATAFRAME=../../data/spleen/dataframe.csv +DEV_IMAGES_DIR=../../../data/spleen/images +DEV_DATAFRAME=../../../data/spleen/dataframe.csv FLIP_PROJECT_ID= FLIP_QUERY= MODEL_CHECKPOINT_URL=https://huggingface.co/aicentreflip/tutorials-evaluation-3d-seg-model/resolve/main/model.pt diff --git a/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/README.md b/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/README.md index 1010028dc..26c4e689d 100644 --- a/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/README.md +++ b/fl-tutorials/nvflare/image_evaluation/3d_spleen_segmentation_evaluation/README.md @@ -41,11 +41,11 @@ The checkpoint URL is configured in `.env.app` as `MODEL_CHECKPOINT_URL`. Default local development settings are in `.env.app`: - `JOB_TYPE=evaluation` -- `DEV_IMAGES_DIR=../../data/spleen/images` -- `DEV_DATAFRAME=../../data/spleen/dataframe.csv` +- `DEV_IMAGES_DIR=../../../data/spleen/images` +- `DEV_DATAFRAME=../../../data/spleen/dataframe.csv` - `MODEL_CHECKPOINT_URL=https://huggingface.co/aicentreflip/tutorials-evaluation-3d-seg-model/resolve/main/model.pt` -The `DEV_*` paths point at the shared, gitignored `fl-tutorials/nvflare/data/spleen` dataset produced by +The `DEV_*` paths point at the shared, gitignored `fl-tutorials/data/spleen` dataset produced by `make -C fl-tutorials download-spleen-data`; `make sim` resolves them to absolute paths so the simulator's client workers find the data. Evaluation settings (e.g. `num_classes`, the `models` checkpoint mapping) are in `app_files/config.json`. diff --git a/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/.env.app b/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/.env.app index 664e88ebd..aaa2de54f 100644 --- a/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/.env.app +++ b/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/.env.app @@ -1,8 +1,8 @@ JOB_TYPE=evaluation -SITE1_IMAGES_DIR=../../data/arkplus/site1_holdoff/accession-resources -SITE1_DATAFRAME=../../data/arkplus/site1_holdoff/sample_get_dataframe_response.csv -SITE2_IMAGES_DIR=../../data/arkplus/site2_holdoff/accession-resources -SITE2_DATAFRAME=../../data/arkplus/site2_holdoff/sample_get_dataframe_response.csv +SITE1_IMAGES_DIR=../../../data/arkplus/site1_holdoff/accession-resources +SITE1_DATAFRAME=../../../data/arkplus/site1_holdoff/sample_get_dataframe_response.csv +SITE2_IMAGES_DIR=../../../data/arkplus/site2_holdoff/accession-resources +SITE2_DATAFRAME=../../../data/arkplus/site2_holdoff/sample_get_dataframe_response.csv RAW_CHECKPOINT=models/Ark6_swinLarge768_ep50.pth.tar FLIP_PROJECT_ID= FLIP_QUERY= diff --git a/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/README.md b/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/README.md index e02f0e6ac..123cee102 100644 --- a/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/README.md +++ b/fl-tutorials/nvflare/image_evaluation/arkplus_baseline_classification_evaluation/README.md @@ -56,7 +56,7 @@ make -C fl-tutorials run-tutorial TUTORIAL=arkplus_baseline_classification_evalu `download-arkplus-eval-data` pulls the `site1_holdoff`/`site2_holdoff` hold-out splits of [`aicentreflip/tutorials-arkplus-cxr-classification`](https://huggingface.co/datasets/aicentreflip/tutorials-arkplus-cxr-classification) -and normalises them into `fl-tutorials/nvflare/data/arkplus/site{1,2}_holdoff/` (gitignored), matching +and normalises them into `fl-tutorials/data/arkplus/site{1,2}_holdoff/` (gitignored), matching this tutorial's `.env.app` defaults. For local development, per-site paths are set in `.env.app`: diff --git a/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/.env.app b/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/.env.app index c3d25b7ac..50cd5214c 100644 --- a/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/.env.app +++ b/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/.env.app @@ -1,7 +1,7 @@ JOB_TYPE=evaluation -SITE1_IMAGES_DIR=../../data/arkplus/site1_holdoff/accession-resources -SITE1_DATAFRAME=../../data/arkplus/site1_holdoff/sample_get_dataframe_response.csv -SITE2_IMAGES_DIR=../../data/arkplus/site2_holdoff/accession-resources -SITE2_DATAFRAME=../../data/arkplus/site2_holdoff/sample_get_dataframe_response.csv +SITE1_IMAGES_DIR=../../../data/arkplus/site1_holdoff/accession-resources +SITE1_DATAFRAME=../../../data/arkplus/site1_holdoff/sample_get_dataframe_response.csv +SITE2_IMAGES_DIR=../../../data/arkplus/site2_holdoff/accession-resources +SITE2_DATAFRAME=../../../data/arkplus/site2_holdoff/sample_get_dataframe_response.csv RAW_CHECKPOINT=models/Ark6_swinLarge768_ep50.pth.tar FINETUNED_CHECKPOINT= diff --git a/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/README.md b/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/README.md index d04a86526..43f8c3828 100644 --- a/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/README.md +++ b/fl-tutorials/nvflare/image_evaluation/arkplus_multimodel_classification_evaluation/README.md @@ -68,7 +68,7 @@ make -C fl-tutorials run-tutorial TUTORIAL=arkplus_multimodel_classification_eva `download-arkplus-eval-data` pulls the `site1_holdoff`/`site2_holdoff` hold-out splits of [`aicentreflip/tutorials-arkplus-cxr-classification`](https://huggingface.co/datasets/aicentreflip/tutorials-arkplus-cxr-classification) -and normalises them into `fl-tutorials/nvflare/data/arkplus/site{1,2}_holdoff/` (gitignored), matching +and normalises them into `fl-tutorials/data/arkplus/site{1,2}_holdoff/` (gitignored), matching this tutorial's `.env.app` defaults (the same hold-out splits the baseline tutorial evaluates). For local development, per-site paths are set in `.env.app`: diff --git a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/.env.app b/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/.env.app index 67ea57ec2..2698316c3 100644 --- a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/.env.app +++ b/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/.env.app @@ -1,6 +1,6 @@ JOB_TYPE=standard -DEV_IMAGES_DIR=../../data/spleen/images -DEV_DATAFRAME=../../data/spleen/dataframe.csv +DEV_IMAGES_DIR=../../../data/spleen/images +DEV_DATAFRAME=../../../data/spleen/dataframe.csv # Any value works for local sim: LOCAL_DEV ignores project_id (data comes from # DEV_DATAFRAME/DEV_IMAGES_DIR) and `make sim` runs the job directly, handing the placeholder # straight to the trainer. Export paths substitute it into the recipe's "--project_id" task arg, diff --git a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/README.md b/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/README.md index 0f4764c50..f5911019b 100644 --- a/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/README.md +++ b/fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation/README.md @@ -26,17 +26,18 @@ upload) step was skipped (the trust PACS supplies CT images only; see ### Dataset setup (local runs) -This directory owns the shared MSD spleen download tooling (`utils/`, with its own `uv` project in -`pyproject.toml`) used by every spleen tutorial. From the repo root: +The shared MSD spleen download tooling used by every spleen tutorial lives in +[`fl-tutorials/datasets/spleen/`](../../../datasets/spleen) (with its own `uv` project in +`pyproject.toml`). From the repo root: ```bash make -C fl-tutorials download-spleen-data # NUM_CASES=<1-41> to control size (default 10) ``` -which runs `utils/download_spleen_dataset.py` (downloads MSD spleen data and reorganises it so each -subject folder holds both the image and its label) followed by `utils/create_spleen_accession_csv.py` -(builds the `accession_id` dataframe the trainer reads in LOCAL_DEV). Data lands under -`fl-tutorials/nvflare/data/spleen/` (gitignored): +which runs `download_spleen_dataset.py` (downloads MSD spleen data and reorganises it so each +subject folder holds both the image and its label) followed by `create_spleen_accession_csv.py` +(builds the `accession_id` dataframe the trainer reads in LOCAL_DEV). Data lands under the shared +`fl-tutorials/data/spleen/` (gitignored): ```text data/spleen/ diff --git a/fl-tutorials/nvflare/image_synthesis/latent_diffusion_model/.env.app b/fl-tutorials/nvflare/image_synthesis/latent_diffusion_model/.env.app index 9a7d4d861..1f4526e82 100644 --- a/fl-tutorials/nvflare/image_synthesis/latent_diffusion_model/.env.app +++ b/fl-tutorials/nvflare/image_synthesis/latent_diffusion_model/.env.app @@ -1,6 +1,6 @@ JOB_TYPE=diffusion_model -DEV_IMAGES_DIR=../../data/spleen/images -DEV_DATAFRAME=../../data/spleen/dataframe.csv +DEV_IMAGES_DIR=../../../data/spleen/images +DEV_DATAFRAME=../../../data/spleen/dataframe.csv FLIP_PROJECT_ID= FLIP_QUERY= diff --git a/fl-tutorials/tests/test_spleen_case_selection.py b/fl-tutorials/tests/test_spleen_case_selection.py index 24cb72723..34bfcf778 100644 --- a/fl-tutorials/tests/test_spleen_case_selection.py +++ b/fl-tutorials/tests/test_spleen_case_selection.py @@ -32,9 +32,7 @@ import pytest TUTORIALS_ROOT = Path(__file__).resolve().parents[1] -DOWNLOADER_PATH = ( - TUTORIALS_ROOT / "nvflare/image_segmentation/3d_spleen_segmentation/utils/download_spleen_dataset.py" -) +DOWNLOADER_PATH = TUTORIALS_ROOT / "datasets/spleen/download_spleen_dataset.py" # The real Task09_Spleen training set: 41 cases, sparsely numbered from 2 to 63. MSD_CASE_NUMBERS = ( diff --git a/flip-api/Makefile b/flip-api/Makefile index 0c7a63627..5301a37cd 100644 --- a/flip-api/Makefile +++ b/flip-api/Makefile @@ -173,14 +173,15 @@ e2e_smoke: # NUM_CASES=41`) and XNAT credentials in the environment (XNAT_USER / XNAT_PASS, or XNAT_HOST too # when not using SPLEEN_XNAT_URLS). The accession-to-case mapping is fetched at run time from the # public aicentreflip/trust-data dataset, so no private repo is required (FLIP#776). -# Absolute paths: the command runs with cwd=SPLEEN_ENRICHMENT_CWD (four levels below the repo -# root), so relative paths written from flip-api/ would not resolve there. -SPLEEN_ENRICHMENT_CWD := ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation +# Absolute paths: the command runs with cwd=SPLEEN_ENRICHMENT_CWD (the shared dataset-tooling +# dir, two levels below the repo root), so relative paths written from flip-api/ would not +# resolve there. +SPLEEN_ENRICHMENT_CWD := ../fl-tutorials/datasets/spleen SPLEEN_XNAT_URLS ?= http://127.0.0.1:8104 http://127.0.0.1:8106 -# The NVFLARE tree by default even though the uploaded app is Flower's: enrichment is -# backend-agnostic, and only the NVFLARE download can reach all 41 mapped cases. -SPLEEN_LABELS_DIR ?= $(abspath ../fl-tutorials/nvflare/data/spleen/images) -SPLEEN_ENRICHMENT_CMD := uv run --no-project --with $(abspath ../flip-utils) python utils/upload_spleen_labels_to_xnat.py --flip-project-id $$FLIP_PROJECT_ID --labels-dir $(SPLEEN_LABELS_DIR) $(foreach url,$(SPLEEN_XNAT_URLS),--xnat-url $(url)) +# The MSD build by default even though the uploaded app is Flower's: enrichment is +# backend-agnostic, and only the MSD download can reach all 41 mapped cases. +SPLEEN_LABELS_DIR ?= $(abspath ../fl-tutorials/data/spleen/images) +SPLEEN_ENRICHMENT_CMD := uv run --no-project --with $(abspath ../flip-utils) python upload_spleen_labels_to_xnat.py --flip-project-id $$FLIP_PROJECT_ID --labels-dir $(SPLEEN_LABELS_DIR) $(foreach url,$(SPLEEN_XNAT_URLS),--xnat-url $(url)) SPLEEN_ENRICHMENT_ARGS := --data-enrichment-cwd $(SPLEEN_ENRICHMENT_CWD) --data-enrichment-cmd '$(SPLEEN_ENRICHMENT_CMD)' e2e_smoke_spleen: MODEL_FILES_DIR := ../fl-tutorials/flower/3d_spleen_segmentation/app diff --git a/flip-api/tests/demo_video.py b/flip-api/tests/demo_video.py index df65d9a0d..4d3310cb4 100644 --- a/flip-api/tests/demo_video.py +++ b/flip-api/tests/demo_video.py @@ -135,14 +135,14 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser.add_argument( "--data-enrichment-cwd", default=None, - help="Directory to run --data-enrichment-cmd in (e.g. the spleen tutorial directory)", + help="Directory to run --data-enrichment-cmd in (e.g. fl-tutorials/datasets/spleen)", ) parser.add_argument( "--data-enrichment-cmd", default=None, help=( "Shell command run OFF-camera between the imaging import and the model segments, with " - "FLIP_PROJECT_ID exported — e.g. the spleen tutorial's utils/upload_spleen_labels_to_xnat.py " + "FLIP_PROJECT_ID exported — e.g. fl-tutorials/datasets/spleen/upload_spleen_labels_to_xnat.py " "(labels must be in place before training)" ), ) diff --git a/flip-ui/test/cypress/demo/README.md b/flip-ui/test/cypress/demo/README.md index 9a1f0467f..215ae3991 100644 --- a/flip-ui/test/cypress/demo/README.md +++ b/flip-ui/test/cypress/demo/README.md @@ -48,8 +48,8 @@ bash scripts/assemble-demo-video.sh test/cypress/demo/videos test/cypress/demo/o # off-camera between the import and the model segments — same enrichment # contract as e2e_smoke, FLIP_PROJECT_ID exported; escape $ once per make): make -C flip-api demo_video DEMO_ARGS="--app spleen --publish-segmentations \ - --data-enrichment-cwd ../fl-tutorials/nvflare/image_segmentation/3d_spleen_segmentation \ - --data-enrichment-cmd 'uv run --no-project --with ../../../../flip-utils python utils/upload_spleen_labels_to_xnat.py --flip-project-id \"\$\$FLIP_PROJECT_ID\" --labels-dir ../../data/spleen/images'" + --data-enrichment-cwd ../fl-tutorials/datasets/spleen \ + --data-enrichment-cmd 'uv run --no-project --with ../../../flip-utils python upload_spleen_labels_to_xnat.py --flip-project-id \"\$\$FLIP_PROJECT_ID\" --labels-dir ../../data/spleen/images'" # → out/flip-demo-spleen.mp4 ```