Troubleshooting

Paths below use ${VG_URL} for your server's base URL and ${VARIANTGRID_DIR} for the install directory (typically /opt/variantgrid, or /mnt/variantgrid on VMs with a small root partition).

General debugging

Handy to see all of Django's output in the console, with python3 manage.py runserver

VCF import hanging

It should usually only take <2 hours for a VCF import and running the annotation pipeline on any new variants. You'll see a hanging import via:

• Sequencing Runs (${VG_URL}/seqauto/) - this page will have a "spinning" logo for ages where the VCF icons usually are...
• Data (${VG_URL}/snpdb/data) - import status of "importing"

There are 3 things that could happen here, and you can tell by the VCF "Vcf import stage". To find this out, click the "VCF" tab on the data page, then click on the link to the VCF that is hung with "importing"

Empty - it failed in the Import process. Click the "View upload processing" - a finished one will have a pie chart, but an unfinished one will just have a grid with a few jobs that are not SUCCESS. Click the "retry import" button. Hopefully it works this time.

Annotating Variants -

See the state of Variant Annotation Runs (${VG_URL}/annotation/variant_annotation_runs) - via menu Annotation -> Variant Annotation Runs link on page.

Annotation runs are leased, and a run whose worker dies stops heartbeating and gets reclaimed automatically within settings.ANNOTATION_RUN_LEASE_SECONDS (default 15 min), up to settings.ANNOTATION_MAX_RUN_ATTEMPTS times before being failed to ERROR. So a genuinely stuck run usually clears itself — check the runs page before intervening.

If runs are in a bad state and you want to reset them so they can be re-dispatched:

python3 manage.py reset_annotation_states

To kick the scheduler (dump unannotated variants and run the pipeline):

python3 manage.py shell

from annotation.tasks.annotation_scheduler_task import annotation_scheduler
annotation_scheduler(active=False)  # runs in background, system stays usable

Annotation run broken

If the celery task is dead but the state isn't errored out, so you can't click "retry upload"

python3 manage.py shell
In [1]: from annotation.models import AnnotationRun

In [2]: ar = AnnotationRun.objects.get(pk=2722)

In [3]: ar.error_exception = "blah"

In [4]: ar.save()

calculating sample stats -

This is a really CPU/Database intensive task, and we run up to 32 of them at a time so sometimes we have Celery jobs crash when the database doesn't allow new connections or something.

Login to server and run management commands (see below)

python3 manage.py calculate_sample_stats

Running management commands

# ssh onto server
sudo su variantgrid
cd ${VARIANTGRID_DIR}
source .venv/bin/activate  # if you're using a venv - see [[Install Python venv]]
python3 manage.py # This will show you all of the commands you can run.

Deleting everything and starting again

• On VCF page, click Sharing/Permissions tab, then delete

This should delete the project, which will be uploaded again. You can wait for a max of 2 hours for this to happen, or go to the sequencing page, click "manage disk scans" then trigger it manually.

If it doesn't re-load the project, try deleting the SequencingRun (click link, then "Admin" then delete) - this should reload everything.

Jobs not running

Go to the Server Status page, ${VG_URL}/variantopedia/server_status (Settings -> Server Status if you're an admin user).

The celery workers should be in green, if they are in red something is wrong and Celery has crashed. In theory the service should restart, but if not try:

sudo ${VARIANTGRID_DIR}/scripts/stop_services.sh
# wait a while
# maybe check ps aux | grep variant - there should be nothing running except the grep command
sudo ${VARIANTGRID_DIR}/scripts/start_services.sh

Server Down

• If the server gets reset due to power etc, it should come back up with the services running, but if it was down long enough, perhaps the IP address will have changed.

If the services aren't running, see above to start them.

Database

To see a list of running processes in the database:

#!bash
sudo su postgres -c 'psql -d snpdb'

Then run SQL:

#!sql
-- To see what queries are running and their PIDs
SELECT * FROM pg_stat_activity;

-- To kill something
SELECT pg_cancel_backend(PID);

-- To REALLY kill something
SELECT pg_terminate_backend(PID);

to kill everything

SELECT pg_cancel_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE datname = current_database()
  AND pid <> pg_backend_pid();

If all else fails

See if you can see any errors here:

• Server Status - ${VG_URL}/variantopedia/server_status
• Event Log - ${VG_URL}/eventlog/

Copy the logs and raise an issue (see Raising Issues):

mkdir vg_logs
scp -r your_user@your_server:/var/log/variantgrid vg_logs
tar cvzf vg_logs.tar.gz vg_logs

Redis

If you get redis errors with "Read Only Filesystem" - you need to add the redis dir to the SystemD service - /etc/systemd/system/redis.service, eg:

ReadWriteDirectories=-/mnt/redis_database

Redis can't save

(error) MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk.

Disable save, then do whatever to clean it (redis-cli flushall, or celery purge --app variantgrid) then allow saving again

127.0.0.1:6379> config get save
1) "save"
2) "900 1 300 10 60 10000"
127.0.0.1:6379> config set save ""
OK
127.0.0.1:6379> config set save "900 1 300 10 60 10000"
OK
127.0.0.1:6379> config get save
1) "save"
2) "900 1 300 10 60 10000"

Celery

Value: 'int' object has no attribute 'signature'.

as per:

Error: This file failed to import due to: Error: File "upload/tasks/vcf/import_vcf_step_task.py", line 131, in schedule_pipeline_stage_steps parallel_tasks.append(task_class.si(upload_step.pk, 0)) File ".../celery/app/task.py", line 784, in si return self.signature(args, kwargs, immutable=True) Type: <class 'AttributeError'>, Value: 'int' object has no attribute 'signature'.

This is caused by not registering the Celery Task class, you need to do eg:

ClassificationImportLinkVariantsTask = app.register_task(ClassificationImportLinkVariantsTask())

Clearing Celery Queues

You need to stop the workers first or you can't purge properly

The queues are defined in variantgrid/settings/components/celery_settings.py: analysis_workers, annotation_workers, db_workers (default), web_workers, scheduling_single_worker and variant_id_single_worker.

To clear just 1 queue:

celery -A variantgrid amqp queue.purge annotation_workers

To clear all the queues:

celery --app variantgrid purge