Skip to content

Variant Annotation

Jump to bottom
Dave Lawrence edited this page Jul 30, 2026 · 3 revisions

How VariantGrid gets VEP annotation into the database.

For the annotation content (which columns exist at which version) see Annotation Column Versions. For VEP implementation details see VEP.

VariantAnnotationVersion

A VariantAnnotationVersion (VAV) is one snapshot of "how we annotate", per genome build — VEP version, annotation consortium, columns version, gene annotation release, plugin data. Annotation rows are tied to a VAV, which is what lets a deployment re-annotate onto a new version while the old data stays queryable.

Create one with:

python3 manage.py create_new_variant_annotation_version --genome-build GRCh38

VariantAnnotationVersion.Status is NEW, ACTIVE or HISTORICAL, with a unique constraint of one ACTIVE version per genome build. A new VAV starts as NEW; promoting it to ACTIVE moves the previous ACTIVE version to HISTORICAL.

The scheduler only ever operates on the ACTIVE version, so a NEW version is a safe place to do work before switching over — this is what External Annotation relies on.

AnnotationRangeLock

Variants are annotated in contiguous blocks of Variant.pk. An AnnotationRangeLock records one such block for a VAV — min_variant, max_variant and a count. Splitting by primary key range means the work partitions deterministically: the same database (or a clone of it) with the same batch size produces identical block boundaries, which is what makes annotated VCFs portable between clones.

Batch size comes from settings.ANNOTATION_VEP_BATCH_MIN / ANNOTATION_VEP_BATCH_MAX.

AnnotationRun

An AnnotationRun is one unit of work against one range lock. It tracks the pipeline as dump → annotate → upload, with an AnnotationStatus:

Created → Dump Started → Dump Completed → Annotation Started → Annotation Completed → Upload Started → Finished, plus Error, Deleting, and Awaiting external annotation (parked for External Annotation).

Timing/provenance is stored per stage (dump_start/dump_end, annotation_start/annotation_end, upload_start/upload_end) along with pipeline_command and pipeline_stdout, which is what the Variant Annotation Runs page shows you.

Runs also carry a pipeline_type — STANDARD (small variants) or STRUCTURAL_VARIANT.

Leasing and dead-worker recovery

Because a run can take a long time, the dispatcher (dispatch_annotation_runs) is the single authority that leases a run before launching it:

  • leased_by / lease_expires — a live run renews its own lease on a background heartbeat every settings.ANNOTATION_RUN_LEASE_HEARTBEAT_SECONDS (default 120s). A worker that dies stops heartbeating, and its run is reclaimed once settings.ANNOTATION_RUN_LEASE_SECONDS (default 900s) passes.
  • attempt_count — reclaims are bounded by settings.ANNOTATION_MAX_RUN_ATTEMPTS (default 3), after which the run is failed to Error.
  • task_id — the in-annotate_variants execution lock, preventing two Celery jobs running the same run.

The heartbeat is why the lease window can be short (15 min) even though a structural variant VEP run can take hours — a long run stays leased by heartbeating, not by a generous static window.

settings.ANNOTATION_UPLOAD_WORKER_SLOTS caps how many runs upload concurrently so imports can use the full db_workers pool.

Monitoring

  • Annotation -> Variant Annotation Runs — per-run status, timings, command and stdout
  • Annotation page — anything red needs upgrading

Recovery commands are in Troubleshooting.

See also

Clone this wiki locally