cdot_rest

REST server for cdot.

We host historical versions of RefSeq and Ensembl transcripts (GRCh37, GRCh38 and T2T-CHM13v2.0) to resolve HGVS.

Public instance: https://cdotlib.org

API

See the API documentation (served from /static/api-docs.html).

Transcript data is stored in Redis. The front page shows the loaded cdot data release.

Loading data

Run from the project directory inside the virtual environment:

sudo su cdot
cd /opt/cdot_rest
source .venv/bin/activate

Latest (recommended)

Download the latest cdot data release (RefSeq and Ensembl, GRCh37 + GRCh38 + T2T-CHM13v2.0) straight from GitHub and load it. The release version is recorded and shown on the front page:

python3 manage.py import_transcript_json latest

A specific file

Download a file from the cdot releases (or create the data from scratch), then load it, passing the consortium and data version:

python3 manage.py import_transcript_json cdot_json \
    --annotation-consortium=RefSeq --cdot-data-version=0.2.32 \
    cdot-0.2.32.refseq.GRCh38.json.gz

python3 manage.py import_transcript_json cdot_json \
    --annotation-consortium=Ensembl --cdot-data-version=0.2.32 \
    cdot-0.2.32.ensembl.GRCh38.json.gz

The same accession can appear in multiple per-build files (eg a RefSeq transcript in both the GRCh37 and GRCh38 files); the loader merges their genome_builds rather than overwriting, so you can load builds separately.

Because loading is additive, upgrading to a new release on top of an old one leaves stale data behind (accessions dropped in the new release, drifting counts). Pass --clear to flush the Redis db first for a clean reload — it works with either subcommand and clears before loading anything:

python3 manage.py import_transcript_json latest --clear

Automatic updates

A systemd timer checks GitHub daily and loads a new data release when one is published. It flushes Redis and reloads, so the site is down for the load (~20 mins). The release version is stored last, so if a load fails the next run retries it.

With RAM for two full copies of the data (8GB), change --clear to --staging-db=1 in cdot-update.service to avoid the downtime: it loads into a spare Redis db and swaps it live with SWAPDB once complete, and a failed or short load never replaces the live data.

# As root
cp config/cdot-update.service config/cdot-update.timer /lib/systemd/system
systemctl daemon-reload
systemctl enable --now cdot-update.timer

systemctl start --no-block cdot-update   # run a check now, in the background (a load takes ~20 mins)
journalctl -u cdot-update -f             # follow the logs - Ctrl-C to stop watching, the job carries on
systemctl status cdot-update             # "inactive (dead)" + status=0/SUCCESS when finished

Without --no-block, systemctl start waits until the load has finished.

The job fails (shows in systemctl --failed) if a newer data release needs a newer cdot client (a data schema change). Upgrade cdot on the server to pick it up.

To copy files onto the server first:

scp -i ~/.ssh/variantgrid-cloud.pem \
    cdot-0.2.32.refseq.GRCh38.json.gz cdot-0.2.32.ensembl.GRCh38.json.gz \
    ubuntu@cdotlib.org:/data/incoming

Install

sudo bash
apt-get update
apt-get upgrade -y

apt-get install -y git-core redis nginx

# uv (https://docs.astral.sh/uv/) - manages the Python virtual environment
curl -LsSf https://astral.sh/uv/install.sh | sh

# User and source code
SYSTEM_CDOT_USER=cdot
CDOT_REST_INSTALL_DIR=/opt/cdot_rest

id -u ${SYSTEM_CDOT_USER} &>/dev/null || useradd ${SYSTEM_CDOT_USER} --create-home --shell "/bin/bash"
mkdir -p ${CDOT_REST_INSTALL_DIR}
chown ${SYSTEM_CDOT_USER} ${CDOT_REST_INSTALL_DIR}
chgrp ${SYSTEM_CDOT_USER} ${CDOT_REST_INSTALL_DIR}

su ${SYSTEM_CDOT_USER}
CDOT_REST_INSTALL_DIR=/opt/cdot_rest  # Again, for the user
cd ${CDOT_REST_INSTALL_DIR}
if [ ! -e ${CDOT_REST_INSTALL_DIR}/.git ]; then
    git clone https://github.com/sacgf/cdot_rest.git .
fi

# Python libraries (gunicorn is included in requirements.txt)
uv venv .venv
source /opt/cdot_rest/.venv/bin/activate
uv pip install -r requirements.txt

Note: requirements.txt currently pins cdot to its git main branch — the latest loader needs get_latest_combo_file_urls / get_latest_data_version_and_release, which are not yet in a PyPI release. Re-pin to a PyPI version once one is published.

Services

# As root
mkdir /var/log/cdot_rest
chown cdot /var/log/cdot_rest
mkdir /run/gunicorn
chown cdot /run/gunicorn/

cp config/gunicorn.service /lib/systemd/system
systemctl enable gunicorn
systemctl start gunicorn

nginx terminates TLS and proxies to gunicorn — see config/nginx.conf.

Also install the automatic updates timer.

Updating a deployed server

sudo su cdot
cd /opt/cdot_rest
source .venv/bin/activate
git pull
uv pip install -r requirements.txt
# nginx serves static files (and llms.txt, robots.txt etc) from cdot_rest/sitestatic
python3 manage.py collectstatic --noinput
exit

# As root
systemctl reload gunicorn

If config/ has changed, copy the changed files into place (/lib/systemd/system for the systemd units, then systemctl daemon-reload; nginx config then nginx -t && systemctl reload nginx).

Development

uv pip install -r requirements-test.txt
python3 manage.py test