Skip to content

Annotation Column Versions

Jump to bottom
Dave Lawrence edited this page Jul 30, 2026 · 16 revisions

Background

Annotation data (VEP version, plugin data files, dbNSFP release etc) is upgraded independently on different deployments. "Columns version" is the knob that lets one codebase serve deployments with different annotation data on disk.

It is driven per genome build by:

settings.ANNOTATION["GRCh37"]["columns_version"] = 5

This controls which columns VariantGrid asks VEP for, and therefore how the VEP command line is built.

Where the column definitions live

Column definitions used to be ColumnVEPField database rows populated by data migrations. That model was deleted (annotation/migrations/0129_delete_columnvepfield.py) — the definitions are now code, in annotation/vep_columns.py:

  • VEPColumnDef — a frozen dataclass mapping a VEP CSQ / plugin / custom output field to one or more VariantGridColumn destinations.
  • VEP_COLUMNS — the tuple of every definition.
  • Each def gates itself with min_columns_version / max_columns_version (and min_vep_version / max_vep_version where a column needs a particular VEP release).
  • VEPColumnDef.applies_to() decides whether a column is in play for a given columns version, VEP version, genome build and pipeline type; has_data_files() drops a column when the matching vep_config data file isn't configured.

annotation/vep_annotation.get_vep_command() reads these to assemble the command line.

Adding a column is now a code change plus a model migration — see Variant Annotation - adding new columns.

Version history

Columns Version 1

Legacy, pre-dating columns_version 2 (ie what VG3 used). dbNSFP "most damaging" fields.

Columns Version 2

May 2022. New pathogenicity prediction tools, dbNSFP rankscores.

Columns Version 3

December 2023, VEP 110.

Added: alphamissense_rankscore, gnomad_faf95, gnomad_faf99, gnomad_fafmax_faf95_max, gnomad_fafmax_faf99_max, gnomad_mid_af, gnomad_non_par, gnomad_xy_ac, gnomad_xy_af, gnomad_xy_an, gnomad_hemi_count, mavedb_score, mavedb_urn. gnomAD v4 became the GRCh38 frequency source.

Columns Version 4

VEP 115. The "raw scores" release — see issue #1625.

Canonical summary (from annotation/tests/test_annotation_vcf.py): columns_version 4 = VEP 115 + dbNSFP 5.3.1a + gnomAD 4.1 + masked SpliceAI + MaveDB 2026-04-30.

  • dbNSFP 4.5a → 5.3.1a, and pathogenicity predictions moved from rankscores to raw scores. The old rankscore columns are capped at max_columns_version=3; the raw-score columns are gated min_columns_version=4. Rankscores are hidden by default on new deployments via settings.ANNOTATION_SHOW_LEGACY_RANKSCORES = False — deployments that already had rankscore data set it True (a value that was already populated is always displayed regardless of the flag).
  • SpliceAI raw → masked scores.
  • gnomAD 4.0 → 4.1 on GRCh38.
  • denovo-db added as a --custom source (denovo_db_pubmed_ids, denovo_db_primary_phenotypes, denovo_db_case_count, denovo_db_control_count).
  • gerp_pp_rs and interpro_domain re-sourced from dbNSFP 5.x.
  • MaveDB data updated to the 2026-04-30 release.
  • Conservation bigwig _max columns (these are gated on min_vep_version=112 rather than on columns_version).

Columns Version 5

VEP 116. New plugins — see issue #1638.

  • ProtVar (all builds) — protvar_stability, protvar_pocket, protvar_int
  • Open Targets (GRCh38) — open_targets_gwas_l2g_score, open_targets_gwas_gene_id, open_targets_gwas_diseases, open_targets_study_type, open_targets_study_id, open_targets_variant_id, open_targets_qtl_gene_id, open_targets_qtl_biosample
  • EVE / popEVE (GRCh38, VEP >= 116) — eve_score, eve_class, popeve_score
  • PromoterAI (GRCh38, VEP >= 116) — promoter_ai_score, promoter_ai_tss_pos

Schema migration: annotation/migrations/0156_vep116_plugin_fields.py.

Configuring your deployment

The package default ANNOTATION in variantgrid/settings/components/annotation_settings.py ships the latest annotation (currently columns_version 5, ANNOTATION_VEP_VERSION = "116"), so a new deployment that has downloaded current annotation data needs no annotation settings at all.

Deployments that haven't loaded the newer data pin themselves back from their variantgrid/settings/env/<hostname>.py. Two helpers are provided, and the caller is also responsible for keeping ANNOTATION_VEP_VERSION on its historical value:

# Have the columns_version 4 data but not VEP 116 + its plugin data
ANNOTATION_VEP_VERSION = "115"
ANNOTATION_VEP_VERSION_DIR = os.path.join(ANNOTATION_VEP_BASE_DIR, "vep_code", ANNOTATION_VEP_VERSION)
ANNOTATION_VEP_CODE_DIR = os.path.join(ANNOTATION_VEP_VERSION_DIR, "ensembl-vep")
ANNOTATION_VEP_PLUGINS_DIR = os.path.join(ANNOTATION_VEP_VERSION_DIR, "plugins")
pin_annotation_to_columns_version_4(ANNOTATION)
# Only have the older columns_version 3 data
ANNOTATION_VEP_VERSION = "110"
ANNOTATION_VEP_VERSION_DIR = os.path.join(ANNOTATION_VEP_BASE_DIR, "vep_code", ANNOTATION_VEP_VERSION)
ANNOTATION_VEP_CODE_DIR = os.path.join(ANNOTATION_VEP_VERSION_DIR, "ensembl-vep")
ANNOTATION_VEP_PLUGINS_DIR = os.path.join(ANNOTATION_VEP_VERSION_DIR, "plugins")
pin_annotation_to_columns_version_3(ANNOTATION)

See variantgrid/settings/env/_settings_template.py for the copy/paste version, and env/shariantcommon.py / env/runx1db2.py for live examples.

Individual builds can sit on different versions — the packaged T2T-CHM13v2.0 config is columns_version 3 while GRCh37/GRCh38 are on 5.

Upgrading to a new columns version

  1. Install the VEP version for the target columns version (Install VEP) and download the annotation data (Install Annotation).

  2. Check the data files are all present:

    python3 manage.py vep_data_check

    To verify file contents where a checksum manifest exists, go to the build's annotation data directory (something like cd /data/annotation/VEP/annotation_data) and run the matching file from ${VARIANTGRID_DIR}/annotation/annotation_data/md5sum_check/, eg:

    md5sum -c ${VARIANTGRID_DIR}/annotation/annotation_data/md5sum_check/vep_110_columns_version3_md5sum.txt

  3. Pull / migrate, then update transcripts:

    python3 manage.py import_cdot_latest

  4. Raise (or remove the pin on) columns_version and ANNOTATION_VEP_VERSION in your settings file.

  5. Confirm VEP itself runs:

    python3 manage.py vep_version --genome-build=GRCh37

  6. Re-annotate. Create a new annotation version and let the scheduler work through it in the background while the system stays usable:

    from annotation.tasks.annotation_scheduler_task import annotation_scheduler
annotation_scheduler(active=False)

    For large re-annotations you can run VEP on another machine entirely — see External Annotation.

Gene annotation release

A columns/VEP upgrade usually wants a matching GeneAnnotationRelease so gene/symbol mappings stay consistent. See Gene and Transcript Version Data for the current commands, then in Django Admin open variant_annotation_version and set gene_annotation_release to what you loaded.

Gene-level annotation from dbNSFP:

python3 manage.py import_dbnsfp_gene_annotation --dbnsfp-version=5.3 /data/annotation/incoming/dbNSFP5.3_gene.complete.gz

Ontology:

python3 manage.py ontology_import --hgnc_sync --mondo ${OD}/mondo.json --hpo ${OD}/hp.owl --biomart ${OD}/mart_export.txt --phenotype_to_genes ${OD}/phenotype_to_genes.txt --gencc ${OD}/gencc-submissions.csv

Then in Admin create a new gene annotation version using the latest data for each build, and check the latest annotation version picked it up (it should be auto-linked).

See also

Clone this wiki locally