Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions .github/workflows/docker_build_monailabel.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
# Copyright (c) Guy's and St Thomas' NHS Foundation Trust & King's College London
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Build and Push Docker Image for MONAI Label

# Direct push trigger like orthanc / xnat_* — monailabel has no test suite to
# gate on. workflow_dispatch covers branch-pinned testing (see CLAUDE.md
# "Docker image builds").
on:
workflow_dispatch:
push:
branches: [main, develop]
paths:
- "trust/monailabel/**"
- ".github/workflows/docker_build_monailabel.yml"
# PR runs build + the import smoke only — the push steps are gated below,
# so nothing is published from a pull request.
pull_request:
paths:
- "trust/monailabel/**"
- ".github/workflows/docker_build_monailabel.yml"

permissions:
contents: read

jobs:
build-and-push:
# Skip on forks: they cannot push to ghcr.io/londonaicentre.
if: github.repository == 'londonaicentre/FLIP'
runs-on: ubuntu-latest
permissions: # override top-level read-only default to allow GHCR push
contents: read
packages: write
defaults:
run:
working-directory: ./trust/monailabel
env:
REGISTRY: ghcr.io
IMAGE_NAME: londonaicentre/monailabel
BUILD_ARGS: ""
steps:
- name: Checkout code
uses: actions/checkout@v5
with:
repository: ${{ github.event.workflow_run.head_repository.full_name }}
ref: ${{ github.event.workflow_run.head_sha }}

# The image is ~10.4 GB (torch + CUDA runtime wheels + SAM2); a stock
# ubuntu-latest runner has ~14 GB free, which the build blows through in
# intermediate layers. Reclaim the ~25 GB of preinstalled toolchains this
# job never uses. Plain rm rather than a third-party action so the
# workflow keeps the repo's actions/* -only supply-chain posture.
- name: Reclaim runner disk space
working-directory: /
run: |
df -h / | tail -1
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
/opt/hostedtoolcache/CodeQL /usr/local/.ghcup /usr/local/share/boost
sudo docker system prune -af --volumes >/dev/null 2>&1 || true
df -h / | tail -1

- name: Determine tags
id: tags
env:
GH_REF_NAME: ${{ github.ref_name }}
GH_EVENT_NAME: ${{ github.event_name }}
GH_REF: ${{ github.ref }}
GH_SHA: ${{ github.sha }}
GH_WR_BRANCH: ${{ github.event.workflow_run.head_branch }}
GH_WR_EVENT: ${{ github.event.workflow_run.event }}
run: |
TAGS="${REGISTRY}/${IMAGE_NAME}:${{ github.sha }}"

# An empty sha would silently publish a mutable literal `sha-` tag — refuse.
[[ -n "$GH_SHA" ]] || { echo "::error::empty commit SHA — cannot compute the sha-<short7> tag"; exit 1; }

# Immutable short-SHA tag (FLIP#751), pushed uniformly on every publish.
# Length 7 must match the tag resolution in deploy/providers/AWS/Makefile.
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:sha-${GH_SHA:0:7}"

# Branch name sanitization
SAFE_REF_NAME=$(echo "$GH_REF_NAME" | sed 's/[^a-zA-Z0-9]/-/g')
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:${SAFE_REF_NAME}"

# Branch number (if starts with number)
if [[ "$GH_REF_NAME" =~ ^[0-9]+ ]]; then
BRANCH_NUM=$(echo "$GH_REF_NAME" | grep -oE '^[0-9]+')
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:${BRANCH_NUM}"
fi

# PR Number (if PR)
if [[ "$GH_EVENT_NAME" == "pull_request" ]]; then
PR_NUMBER=$(echo "$GH_REF" | awk -F / '{print $3}')
TAGS="${TAGS},${REGISTRY}/${IMAGE_NAME}:pr-${PR_NUMBER}"
fi

# Determine if this is a merge/push to main or develop
BRANCH_NAME=""
if [[ "$GH_EVENT_NAME" == "workflow_run" && "$GH_WR_EVENT" == "push" ]]; then
BRANCH_NAME="$GH_WR_BRANCH"
elif [[ "$GH_EVENT_NAME" == "push" ]]; then
BRANCH_NAME="$GH_REF_NAME"
fi

if [[ "$BRANCH_NAME" == "main" ]]; then
TAGS="${TAGS},${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:prod"
elif [[ "$BRANCH_NAME" == "develop" ]]; then
TAGS="${TAGS},${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:stag"
fi
echo "tags=${TAGS}" >> $GITHUB_OUTPUT
echo "Generated tags: ${TAGS}"

- name: Log in to GitHub Container Registry
if: github.event_name != 'pull_request'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_ACTOR: ${{ github.actor }}
run: echo "$GH_TOKEN" | docker login $REGISTRY -u "$GH_ACTOR" --password-stdin

# The Dockerfile itself asserts the two known-bad states at build time
# (unpatched XNAT datastore auth, non-CUDA-12 torch), so a green build
# already guarantees those.
- name: Build Docker image
env:
DOCKER_TAGS: ${{ steps.tags.outputs.tags }}
run: |
IFS=',' read -ra TAG_ARRAY <<< "$DOCKER_TAGS"
TAG_FLAGS=()
for tag in "${TAG_ARRAY[@]}"; do
TAG_FLAGS+=("-t" "$tag")
done
# shellcheck disable=SC2086
docker build $BUILD_ARGS "${TAG_FLAGS[@]}" .

# No GPU on the runner, so this stays at import level: the stack that
# must coexist (monailabel @ pinned commit, torch cu12, sam2, curl for
# the entrypoint's probes) actually loads in the built image.
- name: Smoke test — runtime stack imports
env:
DOCKER_TAGS: ${{ steps.tags.outputs.tags }}
run: |
TAG="${DOCKER_TAGS%%,*}"
docker run --rm --entrypoint sh "$TAG" -c '
curl --version >/dev/null &&
python -c "
import monailabel, torch, sam2
from monailabel.datastore.xnat import XNATDatastore
print(\"monailabel\", monailabel.__version__, \"torch\", torch.__version__)
"'

- name: Push Docker image
if: github.event_name != 'pull_request'
env:
DOCKER_TAGS: ${{ steps.tags.outputs.tags }}
run: |
IFS=',' read -ra TAG_ARRAY <<< "$DOCKER_TAGS"
for tag in "${TAG_ARRAY[@]}"; do
docker push "$tag"
done
20 changes: 20 additions & 0 deletions .github/workflows/test_helm_chart.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,26 @@ jobs:
--set omopDb.external.host=test.example.com \
> /dev/null

# monailabel.enabled defaults false, so every other render skips its whole body —
# render coverage only: the kind install job must NOT enable it (the image is
# ~10.4 GB and needs a GPU, neither of which kind has). Also asserts the
# publicUrl `required` guard actually fires, since it is the one setting that
# cannot be defaulted (the clinician's browser calls it — see trust/README.md).
- name: Render template (monailabel enabled)
run: |
helm template trust-release deploy/providers/kubernetes/ \
--set monailabel.enabled=true \
--set monailabel.publicUrl=http://node.example.com:30030 > /tmp/monailabel.yaml
if ! grep -q "flip-trust.*-monailabel" /tmp/monailabel.yaml; then
echo "::error::monailabel resources did not render with monailabel.enabled=true"
exit 1
fi
if helm template trust-release deploy/providers/kubernetes/ \
--set monailabel.enabled=true > /dev/null 2>&1; then
echo "::error::monailabel rendered without publicUrl — the required guard is gone"
exit 1
fi

# omopDb.vocabLoad.s3Bucket defaults to "" (the licensed bundle has no public
# mirror — FLIP#842/843), so every other render in this job skips the
# vocab-load Job. Without this step its ~110-line body is never rendered in
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ GitHub Actions: `test_flip_api.yml`, `test_flip_ui.yml`, `test_trust_*.yml`, `do

### Docker image builds: gated on tests, manual trigger for branches

**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `flip-ui` is a CI smoke test that never publishes.)
**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*`, `monailabel` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `monailabel` likewise runs an in-job import smoke and reclaims runner disk first, its image being ~10.4 GB against ubuntu-latest's ~14 GB free; `flip-ui` is a CI smoke test that never publishes.)

Every publish also pushes an immutable **`sha-<short7>`** tag (first 7 chars of the built commit) alongside the mutable `:stag`/`:prod` tags. Hub ECS deploys pin these sha tags via task-definition revisions — `make deploy-centralhub` resolves the env branch tip's tag, `make rollback-centralhub` repoints at the previous revision (FLIP#751; see `deploy/providers/AWS/README.md` "Central Hub deploys and rollback"). `deploy-centralhub` also prints an **FL quiesce reminder** (FLIP#770; on `PROD=true` it adds an interactive are-you-sure confirmation, stag stays non-interactive): replacing `fl-server-net-1` kills any in-flight training run, so enable deployment mode first — it pauses FL job pickup (queued jobs hold; the running job finishes and frees its net) — and wait until the hub's `GET /fl/quiesce` reports deployment mode ON and no BUSY net, making "enable mode → wait → deploy → disable" the standard redeploy workflow.

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,7 @@ GitHub Actions: `test_flip_api.yml`, `test_flip_ui.yml`, `test_trust_*.yml`, `do

### Docker image builds: gated on tests, manual trigger for branches

**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `flip-ui` is a CI smoke test that never publishes.)
**The application `docker_build_*.yml` workflows (`flip_api`, `trust_trust_api`, `trust_imaging_api`, `trust_data_access_api`, `omop_db`) auto-publish to GHCR only after their service's test workflow passes on `develop` or `main`.** They trigger via `workflow_run` on the matching test workflow (`FLIP API CI`, `Trust - Trust API CI`, etc.) and a job-level `if` gates on `workflow_run.conclusion == 'success'` — a red test suite never publishes. Path filtering is inherited from the test workflow, so a build still only fires when that service changed. (`orthanc`, `xnat_*`, `monailabel` keep their direct push trigger — they have no separate test workflow to gate on; `orthanc` instead runs an in-job auth smoke test between build and push, and also on PRs touching `trust/orthanc/**`, so a red smoke never publishes — FLIP-PT-091; `monailabel` likewise runs an in-job import smoke and reclaims runner disk first, its image being ~10.4 GB against ubuntu-latest's ~14 GB free; `flip-ui` is a CI smoke test that never publishes.)

Every publish also pushes an immutable **`sha-<short7>`** tag (first 7 chars of the built commit) alongside the mutable `:stag`/`:prod` tags. Hub ECS deploys pin these sha tags via task-definition revisions — `make deploy-centralhub` resolves the env branch tip's tag, `make rollback-centralhub` repoints at the previous revision (FLIP#751; see `deploy/providers/AWS/README.md` "Central Hub deploys and rollback"). `deploy-centralhub` also prints an **FL quiesce reminder** (FLIP#770; on `PROD=true` it adds an interactive are-you-sure confirmation, stag stays non-interactive): replacing `fl-server-net-1` kills any in-flight training run, so enable deployment mode first — it pauses FL job pickup (queued jobs hold; the running job finishes and frees its net) — and wait until the hub's `GET /fl/quiesce` reports deployment mode ON and no BUSY net, making "enable mode → wait → deploy → disable" the standard redeploy workflow.

Expand Down
28 changes: 28 additions & 0 deletions deploy/providers/kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,34 @@ Available services:
| `observability.loki` | Log aggregation | Yes |
| `observability.alloy` | Log collection agent | No (DaemonSet) |
| `observability.grafana` | Metrics dashboard | Yes |
| `monailabel` | Optional AI-assisted annotation in the XNAT OHIF viewer (off by default) | Yes |

### MONAI Label (optional)

Off by default — needs an NVIDIA GPU node and a ~10.4 GB image. Enable with:

```yaml
monailabel:
enabled: true
publicUrl: "http://<node-ip>:30030" # REQUIRED: the clinician's BROWSER calls this —
# XNAT stores it and never proxies it
```

Chart-specific notes (the full operational guide, including the stock-viewer quirks, is
[trust/README.md#monai-label-optional](../../../trust/README.md#monai-label-optional)):

- Reads DICOM straight off xnat-web's archive PVC (read-only `archive` subPath). With the
default `ReadWriteOnce` storage the pod is pinned to xnat-web's node
(`coScheduleWithXnat: true`); only set it `false` on an RWX storage class.
- Exposed as a `NodePort` (default `30030`) so the browser can reach it; a scoped
NetworkPolicy opens exactly that port through the namespace's default-deny ingress
(`service.allowExternalIngress`). The API is unauthenticated — restrict node-port reach
at the network layer.
- Pretrained weights (incl. the ~900 MB SAM checkpoint) persist in the
`monailabel-models` PVC; first start on a cold volume takes minutes (the startup probe
allows 30).
- The chart's XNAT already ships the `ohif-viewer` plugin in its roster (`xnat.web.plugins`),
so no extra plugin step — unlike the compose trust, which deliberately excludes it (FLIP#662).

### External Service Override

Expand Down
Loading