Historical GRCh38 RefSeq alignments - #51 - #124
Merged
Conversation
Add NCBI's RefSeq_historical_alignments (RS_2024_08 set) as a GRCh38 refseq source: 40,885 transcript versions not in the current data (39,566 NM_, 1,319 NR_), mostly old clinical-era versions replaced before any annotation release cdot ingests. The source is two files (annotation + alignments) concatenated before parsing, so the Snakefile gains generic multi-URL source support (urls: list in cdot_transcripts.yaml) and per-source cdot_args (here --skip-missing-parents, for alignments whose transcript has no annotation entry). Merged after UTA and before all official releases, so official annotation always wins. The 2023 blockers are resolved: the duplicated exons came from the old shell script cat-ing the annotation twice, and the short/offset cDNA coordinates are genuine partial alignments (eg NM_000016.3 is 2454 nt but only 1..2423 aligns; NM_002521.1 starts at base 2), which the parser now represents faithfully since the #123 coordinate work.
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Written by Claude
Closes #51 (code side; the data lands with the next data release).
Adds NCBI's historical GRCh38 transcript alignments (
RefSeq_historical_alignments, RS_2024_08 set fromftp.ncbi.nlm.nih.gov/refseq/H_sapiens/historical/GRCh38/) as a refseq/GRCh38 source. Compared against the currentcdot-0.2.33.refseq.GRCh38.json.gzthis contributes 40,885 new transcript versions (39,566 NM_, 1,319 NR_), mostly old clinical-era versions (egNM_000014.3,NM_000016.2) replaced before any annotation release cdot ingests.Pipeline changes
cdot_transcripts.yaml: new source placed after UTA and before the official releases, so anything from an official annotation overwrites it (the ordering decided in Historical GRCh38 refseq #51 back in 2023).Snakefile: generic multi-URL source support. Aurls:list is downloaded as parts and concatenated in list order (annotation first, then alignments; concatenated gzip streams are valid gzip), and a per-sourcecdot_args:key passes extra flags tocdot_json.py(here--skip-missing-parents). Verified with a snakemake dry-run.Why this stalled in 2023, and why it is fine now
NM_000016.3is 2454 nt but only bases 1..2423 align (unaligned poly-A tail), andNM_002521.1's alignment starts at base 2. Since the start_codon/stop_codon: documented 1-based, and undefined across an inter-exon transcript gap #123 coordinate work the parser represents these faithfully; only positions inside an unaligned region cannot be projected.Tests
New fixture
refseq_test.historical_RS_2024_08.gffcut from the real NCBI files (both ACADM historical versions, gapped alignments on both strands, a partial-start case, and an alignment-only orphan) plustest_refseq_gff3_historicalcovering no-duplicate-exons, Gap passthrough, CDS+alignment combination, partial starts andskip_missing_parentsbehaviour. Full suite: 381 passed.🤖 Generated with Claude Code
https://claude.ai/code/session_0152Ksi6d76CwsAooS7dBXVp