Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
26b6399
Add optional MONAI Label server to the trust stack (#55)
atriaybagur Aug 12, 2026
d5670e5
Add export_mask container-service config and document the OHIF viewer…
atriaybagur Aug 12, 2026
68f46f1
Trim the MONAI Label viewer-setup docs back to a paragraph
atriaybagur Aug 12, 2026
d96559f
Fix the monailabel image build: add curl, correct the CUDA assert (#55)
atriaybagur Aug 12, 2026
8850eb7
Document that SAM loads independently of MONAI_LABEL_MODELS
atriaybagur Aug 12, 2026
fc25fb0
Document the viewer's stale-state 'Empty mask' failure and its remedy
atriaybagur Aug 12, 2026
9bd0765
Document the DeepGrow first-segment routing defect in the stock viewer
atriaybagur Aug 12, 2026
8a885ab
Merge remote-tracking branch 'origin/develop' into 55-feature-add-mon…
atriaybagur Aug 12, 2026
f3617b8
Publish the monailabel image and support MONAI Label in production co…
atriaybagur Aug 12, 2026
aa0c67f
Add optional MONAI Label deployment to the Kubernetes trust chart (#948)
atriaybagur Aug 12, 2026
54c1caf
Merge pull request #950 from londonaicentre/948-k8s-monailabel-deploy…
atriaybagur Aug 12, 2026
9dc5948
Merge pull request #949 from londonaicentre/947-publish-monailabel-im…
atriaybagur Aug 12, 2026
5b2c528
Bump the chart's OHIF viewer plugin to 3.8.0 for XNAT 1.10
atriaybagur Aug 12, 2026
9741551
Probe /openapi.json, not /info/ — /info/ walks the whole XNAT datastore
atriaybagur Aug 12, 2026
382fed7
Probe /openapi.json in the entrypoint too, and forward SIGTERM
atriaybagur Aug 13, 2026
34e2fa2
Gate export_mask registration on MONAI_LABEL, and validate it resolved
atriaybagur Aug 13, 2026
fb461bf
Install the ohif-viewer plugin when MONAI Label is enabled
atriaybagur Aug 13, 2026
e0d3d38
Bind MONAI Label to loopback by default and state the auth posture
atriaybagur Aug 13, 2026
4eb2f26
Fix the chart's dropped GPU limit, apply securityContext, scope ingress
atriaybagur Aug 13, 2026
fc95333
Keep the monailabel overlay on the debug and down paths; refuse it on…
atriaybagur Aug 13, 2026
c29d969
Make the monailabel CI assertions fail for the right reasons
atriaybagur Aug 13, 2026
d6427c0
Document the export_mask pipeline and correct the MONAI Label docs
atriaybagur Aug 13, 2026
20961ba
Merge remote-tracking branch 'origin/develop' into 55-feature-add-mon…
atriaybagur Aug 13, 2026
a943c06
Stop the dcm2niix cleanup deleting every project's live subscription
atriaybagur Aug 14, 2026
2eced56
Remove the dcm2niix subscription cleanup rather than repairing it
atriaybagur Aug 14, 2026
8f0977b
Merge branch 'develop' into 55-feature-add-monailabel-to-ohif-viewer-…
atriaybagur Aug 24, 2026
be8f368
Name the remedy when the ohif-viewer opt-in has no jar to fetch
atriaybagur Aug 14, 2026
d135636
Merge branch 'develop' into 55-feature-add-monailabel-to-ohif-viewer-…
atriaybagur Aug 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions .github/workflows/docker_build_monailabel.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# Copyright (c) Guy's and St Thomas' NHS Foundation Trust & King's College London
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Build and Push Docker Image for MONAI Label

# Direct push trigger like orthanc / xnat_* — monailabel has no test suite to
# gate on. workflow_dispatch covers branch-pinned testing (see CLAUDE.md
# "Docker image builds").
on:
workflow_dispatch:
push:
branches: [main, develop]
paths:
- "trust/monailabel/**"
- ".github/workflows/docker_build_monailabel.yml"
# PR runs build + the import smoke only — the push steps are gated below,
# so nothing is published from a pull request.
pull_request:
paths:
- "trust/monailabel/**"
- ".github/workflows/docker_build_monailabel.yml"

permissions:
contents: read

jobs:
build-and-push:
# Skip on forks: they cannot push to ghcr.io/londonaicentre.
if: github.repository == 'londonaicentre/FLIP'
runs-on: ubuntu-latest
permissions: # override top-level read-only default to allow GHCR push
contents: read
packages: write
defaults:
run:
working-directory: ./trust/monailabel
env:
REGISTRY: ghcr.io
IMAGE_NAME: londonaicentre/monailabel
BUILD_ARGS: ""
steps:
- name: Checkout code
uses: actions/checkout@v5
with:
repository: ${{ github.event.workflow_run.head_repository.full_name }}
ref: ${{ github.event.workflow_run.head_sha }}

# The image is ~10.4 GB (torch + CUDA runtime wheels + SAM2); a stock
# ubuntu-latest runner has ~14 GB free, which the build blows through in
# intermediate layers. Reclaim the ~25 GB of preinstalled toolchains this
# job never uses. Plain rm rather than a third-party action so the
# workflow keeps the repo's actions/* -only supply-chain posture.
- name: Reclaim runner disk space
working-directory: /
run: |
df -h / | tail -1
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
/opt/hostedtoolcache/CodeQL /usr/local/.ghcup /usr/local/share/boost
sudo docker system prune -af --volumes >/dev/null 2>&1 || true
df -h / | tail -1
# Assert the reclaim actually delivered. `rm -rf` on a path that no longer exists
# exits 0, so if a future runner image relocates these toolchains this step stays
# green and the build instead dies with an opaque ENOSPC deep into a torch layer.
# Fail here, where the cause is obvious. Floor is ~2x the finished image.
FREE_GB=$(df -BG --output=avail / | tail -1 | tr -dc '0-9')
echo "Free after reclaim: ${FREE_GB} GB"
if [[ "${FREE_GB}" -lt 25 ]]; then
echo "::error::Only ${FREE_GB} GB free after reclaim; the ~10.4 GB image needs ~25 GB of headroom for intermediate layers. The runner image's preinstalled toolchain paths have probably moved — update this step."
exit 1
fi

- name: Determine tags
id: tags
env:
GH_REF_NAME: ${{ github.ref_name }}
GH_EVENT_NAME: ${{ github.event_name }}
GH_REF: ${{ github.ref }}
GH_SHA: ${{ github.sha }}
GH_WR_BRANCH: ${{ github.event.workflow_run.head_branch }}
GH_WR_EVENT: ${{ github.event.workflow_run.event }}
run: |
TAGS="${REGISTRY}/${IMAGE_NAME}:${{ github.sha }}"

# An empty sha would silently publish a mutable literal `sha-` tag — refuse.
[[ -n "$GH_SHA" ]] || { echo "::error::empty commit SHA — cannot compute the sha-<short7> tag"; exit 1; }

# Immutable short-SHA tag (FLIP#751), pushed uniformly on every publish.
# Length 7 must match the tag resolution in deploy/providers/AWS/Makefile.
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:sha-${GH_SHA:0:7}"

# Branch name sanitization
SAFE_REF_NAME=$(echo "$GH_REF_NAME" | sed 's/[^a-zA-Z0-9]/-/g')
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:${SAFE_REF_NAME}"

# Branch number (if starts with number)
if [[ "$GH_REF_NAME" =~ ^[0-9]+ ]]; then
BRANCH_NUM=$(echo "$GH_REF_NAME" | grep -oE '^[0-9]+')
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:${BRANCH_NUM}"
fi

# PR Number (if PR)
if [[ "$GH_EVENT_NAME" == "pull_request" ]]; then
PR_NUMBER=$(echo "$GH_REF" | awk -F / '{print $3}')
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:pr-${PR_NUMBER}"
fi

# Determine if this is a merge/push to main or develop
BRANCH_NAME=""
if [[ "$GH_EVENT_NAME" == "workflow_run" && "$GH_WR_EVENT" == "push" ]]; then
BRANCH_NAME="$GH_WR_BRANCH"
elif [[ "$GH_EVENT_NAME" == "push" ]]; then
BRANCH_NAME="$GH_REF_NAME"
fi

if [[ "$BRANCH_NAME" == "main" ]]; then
TAGS="${TAGS},${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:prod"
elif [[ "$BRANCH_NAME" == "develop" ]]; then
TAGS="${TAGS},${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:stag"
fi
echo "tags=${TAGS}" >> $GITHUB_OUTPUT
echo "Generated tags: ${TAGS}"

- name: Log in to GitHub Container Registry
if: github.event_name != 'pull_request'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_ACTOR: ${{ github.actor }}
run: echo "$GH_TOKEN" | docker login $REGISTRY -u "$GH_ACTOR" --password-stdin

# The Dockerfile itself asserts the two known-bad states at build time
# (unpatched XNAT datastore auth, non-CUDA-12 torch), so a green build
# already guarantees those.
- name: Build Docker image
env:
DOCKER_TAGS: ${{ steps.tags.outputs.tags }}
run: |
IFS=',' read -ra TAG_ARRAY <<< "$DOCKER_TAGS"
TAG_FLAGS=()
for tag in "${TAG_ARRAY[@]}"; do
TAG_FLAGS+=("-t" "$tag")
done
# shellcheck disable=SC2086
docker build $BUILD_ARGS "${TAG_FLAGS[@]}" .

# No GPU on the runner, so this stays at import level: the stack that
# must coexist (monailabel @ pinned commit, torch cu12, sam2, curl for
# the entrypoint's probes) actually loads in the built image.
- name: Smoke test — runtime stack imports
env:
DOCKER_TAGS: ${{ steps.tags.outputs.tags }}
run: |
TAG="${DOCKER_TAGS%%,*}"
docker run --rm --entrypoint sh "$TAG" -c '
curl --version >/dev/null &&
python -c "
import monailabel, torch, sam2
from monailabel.datastore.xnat import XNATDatastore
print(\"monailabel\", monailabel.__version__, \"torch\", torch.__version__)
"'

- name: Push Docker image
if: github.event_name != 'pull_request'
env:
DOCKER_TAGS: ${{ steps.tags.outputs.tags }}
run: |
IFS=',' read -ra TAG_ARRAY <<< "$DOCKER_TAGS"
for tag in "${TAG_ARRAY[@]}"; do
docker push "$tag"
done
29 changes: 29 additions & 0 deletions .github/workflows/test_helm_chart.yml
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,35 @@ jobs:
--set omopDb.external.host=test.example.com \
> /dev/null

# monailabel.enabled defaults false, so every other render skips its whole body —
# render coverage only: the kind install job must NOT enable it (the image is
# ~10.4 GB and needs a GPU, neither of which kind has). Also asserts the
# publicUrl `required` guard actually fires, since it is the one setting that
# cannot be defaulted (the clinician's browser calls it — see trust/README.md).
- name: Render template (monailabel enabled)
run: |
helm template trust-release deploy/providers/kubernetes/ \
--set monailabel.enabled=true \
--set monailabel.publicUrl=http://node.example.com:30030 > /tmp/monailabel.yaml
if ! grep -q "flip-trust.*-monailabel" /tmp/monailabel.yaml; then
echo "::error::monailabel resources did not render with monailabel.enabled=true"
exit 1
fi
# Assert the guard fires AND fires for the right reason. Testing only "helm exited
# non-zero" would be satisfied by a typo'd chart path, an unrelated schema error, or
# any broken sibling template — a vacuous pass of exactly the kind this workflow
# calls out elsewhere. Capture stderr and match the guard's own message.
if helm template trust-release deploy/providers/kubernetes/ \
--set monailabel.enabled=true > /tmp/monailabel-noguard.out 2>&1; then
echo "::error::monailabel rendered without publicUrl — the required guard is gone"
exit 1
fi
if ! grep -q "monailabel.publicUrl is required" /tmp/monailabel-noguard.out; then
echo "::error::render without publicUrl failed, but not on the publicUrl guard:"
cat /tmp/monailabel-noguard.out
exit 1
fi

# omopDb.vocabLoad.s3Bucket defaults to "" (the licensed bundle has no public
# mirror — FLIP#842/843), so every other render in this job skips the
# vocab-load Job. Without this step its ~110-line body is never rendered in
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ GitHub Actions: `test_flip_api.yml`, `test_flip_ui.yml`, `test_trust_*.yml`, `fl

### Docker image builds: gated on tests, manual trigger for branches

**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `flip-ui` is a CI smoke test that never publishes.)
**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*`, `monailabel` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `monailabel` likewise runs an in-job import smoke and reclaims runner disk first, its image being ~10.4 GB against ubuntu-latest's ~14 GB free; `flip-ui` is a CI smoke test that never publishes.)

Every publish also pushes an immutable **`sha-<short7>`** tag (first 7 chars of the built commit) alongside the mutable `:stag`/`:prod` tags. Hub ECS deploys pin these sha tags via task-definition revisions — `make deploy-centralhub` resolves the env branch tip's tag, `make rollback-centralhub` repoints at the previous revision (FLIP#751; see `deploy/providers/AWS/README.md` "Central Hub deploys and rollback"). `deploy-centralhub` also prints an **FL quiesce reminder** (FLIP#770; on `PROD=true` it adds an interactive are-you-sure confirmation, stag stays non-interactive): replacing `fl-server-net-1` kills any in-flight training run, so enable deployment mode first — it pauses FL job pickup (queued jobs hold; the running job finishes and frees its net) — and wait until the hub's `GET /fl/quiesce` reports deployment mode ON and no BUSY net, making "enable mode → wait → deploy → disable" the standard redeploy workflow.

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ GitHub Actions: `test_flip_api.yml`, `test_flip_ui.yml`, `test_trust_*.yml`, `fl

### Docker image builds: gated on tests, manual trigger for branches

**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `flip-ui` is a CI smoke test that never publishes.)
**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*`, `monailabel` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `monailabel` likewise runs an in-job import smoke and reclaims runner disk first, its image being ~10.4 GB against ubuntu-latest's ~14 GB free; `flip-ui` is a CI smoke test that never publishes.)

Every publish also pushes an immutable **`sha-<short7>`** tag (first 7 chars of the built commit) alongside the mutable `:stag`/`:prod` tags. Hub ECS deploys pin these sha tags via task-definition revisions — `make deploy-centralhub` resolves the env branch tip's tag, `make rollback-centralhub` repoints at the previous revision (FLIP#751; see `deploy/providers/AWS/README.md` "Central Hub deploys and rollback"). `deploy-centralhub` also prints an **FL quiesce reminder** (FLIP#770; on `PROD=true` it adds an interactive are-you-sure confirmation, stag stays non-interactive): replacing `fl-server-net-1` kills any in-flight training run, so enable deployment mode first — it pauses FL job pickup (queued jobs hold; the running job finishes and frees its net) — and wait until the hub's `GET /fl/quiesce` reports deployment mode ON and no BUSY net, making "enable mode → wait → deploy → disable" the standard redeploy workflow.

Expand Down
51 changes: 47 additions & 4 deletions deploy/providers/kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,48 @@ Available services:
| `observability.loki` | Log aggregation | Yes |
| `observability.alloy` | Log collection agent | No (DaemonSet) |
| `observability.grafana` | Metrics dashboard | Yes |
| `monailabel` | Optional AI-assisted annotation in the XNAT OHIF viewer (off by default) | Yes |

### MONAI Label (optional)

Off by default — needs an NVIDIA GPU node and a ~10.4 GB image. Enable with:

```yaml
monailabel:
enabled: true
publicUrl: "http://<node-ip>:30030" # REQUIRED: the clinician's BROWSER calls this —
# XNAT stores it and never proxies it
```

Chart-specific notes (the full operational guide, including the stock-viewer quirks, is
[trust/README.md#monai-label-optional](../../../trust/README.md#monai-label-optional)):

- Reads DICOM straight off xnat-web's archive PVC (read-only `archive` subPath). With the
default `ReadWriteOnce` storage the pod is pinned to xnat-web's node
(`coScheduleWithXnat: true`); only set it `false` on an RWX storage class.
- Exposed as a `NodePort` (default `30030`) so the browser can reach it. When
`service.allowExternalIngress` and `networkPolicies.enabled` are set, a NetworkPolicy opens
the **pod** port (`monailabel.port`, default `8030`) through the namespace's default-deny
ingress — NodePort traffic is DNAT'd to the pod port before policy evaluation, so that is
the port the rule names, not `30030`.
- **Scope that policy with `monailabel.allowedIngressCIDRs`.** Left empty (the default) the
ingress rule has no `from:`, and a NetworkPolicy rule without `from:` matches **every**
source, in and out of cluster. That is the fallback because the chart cannot know where
clinicians browse from, but the API behind it is unauthenticated (`monailabel.authEnable`
is upstream's own switch and needs an OAuth realm FLIP does not run) and holds the XNAT
service-account credentials. Set the subnets your clinicians use, e.g.
`allowedIngressCIDRs: ["10.0.0.0/8"]`.
- Pretrained weights (incl. the ~900 MB SAM checkpoint) persist in the
`<release>-monailabel-models` PVC; first start on a cold volume takes minutes (the startup
probe allows 30).
- The chart's XNAT already ships the `ohif-viewer` plugin in its roster (`xnat.web.plugins`),
so no extra plugin step — unlike the compose trust, which deliberately excludes it (FLIP#662).
- **No `export_mask` converter here.** The compose trust registers a container-service command
that turns saved DICOM-SEG assessors into the NIfTI that FL training consumes
(`trust/xnat/xnat/config/configure-export-mask.sh`, run from `xnat-configure`); the chart's
`xnat-init-job` has no equivalent. So on Kubernetes the annotation UI works but masks are
not auto-converted — the inverse of the compose trust, which has the converter and excludes
the viewer.

### External Service Override

Expand Down Expand Up @@ -469,13 +511,14 @@ old install on the previous chart version.
- **Pod Security & container hardening**: the chart-created namespace
carries Pod Security Standards labels (`enforce=baseline`, `warn`/`audit=restricted`
by default — tune via `podSecurity.*`), and the stateless services
(trust-api, imaging-api, data-access-api, fl-client)
(trust-api, imaging-api, data-access-api, fl-client) plus `monailabel`
apply a container `securityContext` (`allowPrivilegeEscalation: false`, drop
`ALL` capabilities, `seccompProfile: RuntimeDefault`) from `.Values.securityContext`.
`runAsNonRoot` / `readOnlyRootFilesystem` are left opt-in (image-dependent).
**Remaining for full `restricted` enforcement:** the stateful images
(`xnat-web`, `xnat-db`, `omop-db`, `orthanc`) need `fsGroup`/chown init
containers before they can run non-root.
**Remaining for full `restricted` enforcement:** the images that own a
PersistentVolume (`xnat-web`, `xnat-db`, `omop-db`, `orthanc`, and `monailabel`
— whose weights PVC the server writes at model-load time) need `fsGroup`/chown
init containers before they can run non-root.

## Development

Expand Down
Loading
Loading