Skip to content

Upgrading

Jump to bottom
Dave Lawrence edited this page Aug 6, 2026 · 5 revisions

Process

  • Email affected users (2 days in advance)
  • Add a site message with countdown
  • Inform users of fixes and new features via changelog.html
  • Server should be upgraded out of hours (and check that nobody is using system during upgrade)
  • Post upgrade - use site messages to inform users of any information they need to know

Site messages for User Notification

Before upgrading a live site, it's best to warn the users. You can do this via admin (prob easiest to set date) or command line:

python3 manage.py site_messages --shutdown=1440  # 24 hours notice

This will add a message like "The system will soon be shut down soon (4 minutes and 59 seconds from now)" to the top of logged in screens.

After the upgrade, you can then run:

python3 manage.py site_messages --clear-old

Which will remove any messages with dates in the past.

Libraries

New libraries should have been added to the requirements file. Install them into the virtual environment (see Install Python venv) — not system-wide, which recent Ubuntu blocks anyway:

cd ${VARIANTGRID_DIR}
source .venv/bin/activate
uv pip install -r requirements.txt

Upgrade Tool

We have an upgrade utility script in ./scripts/upgrade.sh, which is a thin wrapper around scripts/migrator/migrator.py:

./scripts/upgrade.sh                # interactive menu
./scripts/upgrade.sh --quick        # run the standard steps, quit if nothing else is outstanding
./scripts/upgrade.sh --auto-manage  # run all outstanding, unblocked manage.py steps
./scripts/upgrade.sh --help

The standard steps (menu option a) are:

  • git pull
  • migrate
  • collectstatic_js_reverse
  • collectstatic
  • deployment_check --die-if-invalid
  • deployed (records the deploy in Rollbar)

A failed step stops the run — the following steps don't get attempted.

In addition, if there are manual migration steps required they are listed under ****** SPECIAL STEPS ****** in the menu, numbered from 1, and you run them after the standard steps. Each runs on its own and the menu is then redrawn with the remaining tasks renumbered, so the next one to run is again 1. Menu option am (or --auto-manage) ploughs through all of them that are runnable, stopping on the first failure. Manual steps are either "manage" (as in python manage.py somecommand) or "other" (prompt the user with text to perform a step).

Steps can be tagged in the menu:

  • [BLOCKED] — waiting on a gate; satisfy it with python3 manage.py manual_gate --satisfy <gate> (python3 manage.py manual_gate lists gate status). Selecting it from the menu still runs it.
  • [OBSOLETE] — the command no longer exists in this codebase; selecting it offers to mark it complete.

To create these manual steps, in a migration script:

operations = [
    ManualOperation.operation_manage(["dummy_command"])
]

operation_manage / operation_other take the args as a list, plus optional:

  • note= — text (or a callable taking apps) shown to whoever runs the upgrade
  • test= — a callable taking apps; the task is only registered if this returns True, so deployments with no affected data aren't asked to do anything
  • requires= — task ids that must run first (build them with ManualOperation.task_id_manage(...))
def _needs_fixing(apps):
    Thing = apps.get_model("myapp", "Thing")
    return Thing.objects.filter(broken=True).exists()

operations = [
    ManualOperation.operation_manage(["dummy_command"], test=_needs_fixing)
]

If you want to call it directly in a RunPython

if dummy_command_required:
    ManualOperation.operation_manage(["dummy_command"]).run(apps)

See snpdb/migrations/0188_one_off_migrate_common_filter_gnomad_versions.py for a live example.

Note: The upgrade tool does not restart services (which requires root access) - see below

Services

# Start services
sudo ./scripts/start_services.sh

# Stop Services
sudo ./scripts/stop_services.sh
# maybe check with ps aux | grep variant # nothing should show other than grep command

# Restart services
sudo ./scripts/restart_services.sh
  • Go to the Annotation page (top menu) and look for anything red (annotation that needs upgrading)
  • Go to the Server Status page (Settings->Server Status) to make sure celery workers are running.

Clone this wiki locally