Add propose_clades.py tooling to find clade designation candidates

.claude/skills/designate-clades/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: designate-clades
+description: >-
+  Propose and draft new SARS-CoV-2 Nextstrain clade designations from open data.
+  TRIGGER when the user wants to find/designate/elevate a new clade, check whether
+  any Pango lineage has crossed the clade-naming threshold, refresh clade candidates,
+  or draft a clade-designation PR in the nextstrain/ncov repo. Runs
+  scripts/propose_clades.py, interprets the candidate dossier, writes the five
+  defaults/* edits, validates them, and drafts the PR.
+---
+
+# Designating SARS-CoV-2 Nextstrain clades
+
+Clade designation is a judgement call on top of a quantitative threshold. The
+`scripts/propose_clades.py` tool does the *evidence gathering*; you make the
+*decisions* (which lineage to elevate, where to draw the line, the name, the prose)
+and write the edits. The tool never edits `defaults/` — that is your job here.
+
+## Designation criteria
+
+A candidate Pango lineage qualifies when it **both**:
+1. carries **>=1 spike (S:) mutation** relative to its parent Nextstrain clade, and
+2. has risen to **>30% regional** (North America or Europe) **or >20% global** frequency.
+
+Only ever elevate an **existing Pango lineage** to a clade — keep a strict 1:1
+Pango↔Nextstrain-clade mapping. The hard part is usually *which* lineage in a nested
+chain (e.g. XFG.1 vs XFG.1.1) to elevate: **prefer the branch with the larger fitness
+differential**, visible as a jump in MLR growth advantage and corroborated by the
+frequency trajectory.
+
+Open data is sparse and ~2 months lagged in 2026 (GISAID API access was lost — see the
+2026-02-24 blog). Treat every percentage as approximate and triangulate the three
+frequency reads the dossier gives you (build tip-freq, LAPIS exact, MLR modeled).
+
+## Step 1 — Generate the dossier
+
+```
+python3 scripts/propose_clades.py --data-source open               # global + NA + Europe
+python3 scripts/propose_clades.py --data-source gisaid --group blab # global + all 6 regions
+```
+
+Everything is fetched live (cached under `results/clade_cache/`; `--refresh` to re-pull): the
+Nextclade reference tree, the Nextstrain 6m builds, forecasts-ncov MLR fitness, and — for **open**
+only — LAPIS exact frequencies. **gisaid** auto-discovers the group's latest dated Groups builds, so
+the date is handled for you (`--gisaid-date` pins an older one). open is GISAID-free but
+North-America-dominated; gisaid covers all regions and surfaces sweeps open misses.
+
+Frequency is judged conservatively: a fixed-N=50 window of date-sorted tips slides across a
+~6mo→3wk band and a candidate flags only if the **one-sided 95% Wilson lower bound** clears the
+threshold (>20% global / >30% regional) in some window — so each flag is a *confident* majority
+(effective bar ≈46% regional / ≈34% global at N=50), reported as `peak X% [n, as of <date>, lo Y%] ·
+now Z%`. A geography with <50 band tips reads `insufficient`. So a flag already means the data backs
+it; your job is the demarcation/naming judgement, not re-litigating noise.
+
+Outputs:
+- `results/clade_candidates.md` — human-readable dossier (read this)
+- `results/clade_candidates.json` — same data, structured (paste-ready edits live here)
+
+## Step 2 — Interpret each flagged candidate
+
+For every candidate under "Flagged", read the evidence block and cross-check against the
+live resources before trusting the numbers:
+- **Nextstrain builds:** https://nextstrain.org/ncov/open/global/6m ,
+  `.../north-america/6m` , `.../europe/6m`
+- **Cov-Spectrum** (exact, per lineage): e.g.
+  `https://cov-spectrum.org/explore/World/AllSamples/Past6M/variants?nextcladePangoLineage=<LIN>*`
+- **MLR fitness/forecast:** https://nextstrain.github.io/forecasts-ncov/
+- **Nextclade reference tree** (mutations): https://nextstrain.org/nextclade/nextstrain/sars-cov-2/wuhan-hu-1/orfs
+
+Use WebFetch on the LAPIS open API for an exact recent read if you want a tighter window
+than the dossier's 180 days, e.g.
+`https://lapis.cov-spectrum.org/open/v2/sample/aggregated?nextcladePangoLineage=<LIN>*&region=North%20America&dateFrom=<YYYY-MM-DD>`
+(divide by the same query without the lineage filter for the denominator).
+
+Weigh, since a flag already passed the windowed CI:
+- **only one small region clears** (e.g. a 56% Oceania-only majority) — decide whether it warrants a
+  *global* clade name or is just locally dominant; the per-geography lines and `now` values show reach;
+- the **peak is months old and `now` has collapsed** — still designation-worthy retrospectively, but
+  note it in the PR;
+- the lineage is **recombinant (X*)** — confirm the breakpoint and that the defining mutations are
+  inherited cleanly;
+- for **open**, LAPIS is shown as a secondary exact read; large build-vs-LAPIS gaps are expected
+  (LAPIS is raw/country-biased, the build is population-weighted) — prefer the build for the call.
+
+## Step 3 — Decide the demarcation
+
+Look at the "Where to draw the line" table. Walk the nested chain (parent → candidate →
+children) and pick the lineage where the **MLR growth advantage jumps most** (largest `Δga`
+/ highest `ga`) and whose frequency is actually rising (`trend` > 0). That is the branch
+carrying the real fitness differential. Elevating a parent that is mostly one fast child, or
+a child that is only a slice of an already-rising parent, are both mistakes — the table is
+there to tell those apart.
+
+If the answer is "this is just slicing the current dominant clade finer with no real fitness
+story," it is fine to **not** designate. Note it on the watch list and move on.
+
+## Step 4 — Write the five `defaults/` edits
+
+The dossier prints a paste-ready block per candidate — apply it as-is, keeping **all** the
+defining `clades.tsv` nuc rows (the full net set vs the parent clade). Using the full set makes
+the clade **exactly specific** to the Pango lineage and demarcates precisely at it rather than a
+broader/higher node. The spike rows are flagged so you can cite them for the ≥1-spike criterion
+and the PR; only drop a row for a specific reason (e.g. a known homoplasic/unstable site that
+breaks membership across the lineage).
+
+1. `defaults/clades.tsv` — `<NAME>\tclade\t<PARENT>` + all the `<NAME>\tnuc\t<site>\t<alt>` rows
+2. `defaults/clade_display_names.yml` — `<NAME>: <NAME> (<Pango>)`
+3. `defaults/clade_hierarchy.tsv` — `<NAME>\t<PARENT>\t<WHO>`
+4. `defaults/clade_emergence_dates.tsv` — `<NAME>\t<YYYY-MM-01>` (first-seen from the dossier)
+5. `defaults/color_ordering.tsv` — `clade_membership\t<NAME> (<Pango>)`
+
+The suggested `<NAME>` (e.g. `26A`) uses the **designation year** — the year the clade is being
+named, i.e. the current year — *not* the lineage's emergence year. It is the first unused letter
+for that year (so the first clade named in 2026 is `26A`, the next `26B`, and so on). The
+emergence date still goes in `clade_emergence_dates.tsv` as the lineage's first-seen date. Only
+touch `emerging_lineages.tsv` for recombinant/special lineages, removing a lineage there if you
+are promoting it to a full clade (see PR #1152).
+
+## Step 5 — Validate
+
+Confirm the new definition parses and inherits correctly through augur:
+
+```
+scripts/expand-clade-definitions defaults/clades.tsv | grep "^<NAME>"
+```
+
+(`scripts/expand-clade-definitions` imports `augur.clades.read_in_clade_definitions`; run it
+with a Python that has augur installed.) Expect `<NAME>` to resolve to its inherited parent-clade
+chain plus the new mutations, with no error. The full region build in Step 6 is the real visual
+validation — you see the tips pick up `<NAME>` in Auspice.
+
+## Step 6 — Build a trial view and deploy to staging
+
+Standard practice is to attach a live trial build to a designation PR so reviewers can see the new
+clade in the tree. This is quick — editing `clades.tsv` only reruns the clade-downstream Snakemake
+steps.
+
+1. On the designation branch, rebuild the region(s) where the clade is prominent (the flagging
+   regions from the dossier — at least the strongest, usually plus `global`):
+   ```
+   nextstrain build . --profile nextstrain_profiles/nextstrain-gisaid \
+     auspice/ncov_gisaid_<region>_6m.json        # nextstrain-open for an open designation
+   ```
+2. Rename the rebuilt outputs to a branch-named trial dataset and deploy to staging (needs the
+   nextstrain CLI with deploy access to `s3://nextstrain-staging`):
+   ```
+   for f in auspice/ncov_gisaid_<region>_6m*; do
+     mv "$f" "${f/ncov_gisaid_/ncov_<branch>_}"   # e.g. ncov_designate-26B_<region>_6m*
+   done
+   nextstrain deploy s3://nextstrain-staging auspice/ncov_<branch>_<region>_6m*
+   ```
+   It goes live at `https://nextstrain.org/staging/ncov/<branch>/<region>/6m` (filename `_` → URL
+   `/`). Use the **branch name** as the trial slug. Staging is the standard home for PR-related
+   ephemera. (This is the lightweight local path; the heavier GitHub-Action trial builds are in
+   `docs/dev_docs.md` → "Triggering trial builds".)
+
+## Step 7 — Draft the PR
+
+Open a PR in the house style of #1149 / #1152 / #1158 / #1192 / #1204 / #1206. **Be thorough — not a
+one-line summary.** The justification should state:
+- the new clade name and its Pango lineage, and the parent clade it descends from;
+- the defining **spike mutation(s)** (and, since the definition keeps all defining nuc for
+  specificity, a note to that effect if relevant);
+- a **per-region frequency breakdown** pulled straight from `results/clade_candidates.{md,json}`:
+  for each region the clade is notable in, the **peak %, the n=50 window's one-sided 95% lower
+  bound, the as-of date, and the current %**, plus the global number — so reviewers can trace them;
+- the **MLR fitness** (growth advantage) where available;
+- the **demarcation rationale** (why this lineage vs its parent/child alternatives, citing the
+  mutations that differ, which the dossier lists);
+- the **staging trial link(s)** from Step 6.
+
+Keep the dossier's exact numbers in the PR body so reviewers can trace them. Do not push or
+open the PR without the user's go-ahead.

docs/dev_docs.md

@@ -3,9 +3,11 @@
 ## Contents
 
  1. [Setup](#setup)
- 1. [Data](#data)
- 1. [Running](#running)
- 1. [Releasing new workflow versions](#releasing-new-workflow-versions)
+ 2. [Data](#data)
+ 3. [Running](#running)
+ 4. [Releasing new workflow versions](#releasing-new-workflow-versions)
+ 5. [Proposing new clade designations](#proposing-new-clade-designations)
+ 6. [Running Core Nextstrain Builds](#running-core-nextstrain-builds)
 
 ## Running
 
@@ -24,6 +26,117 @@ Prior to merging a pull request that introduces a new backward incompatible chan
 We do not release new minor versions for new features, but you should document new features in the change log as part of the corresponding pull request under a heading for the date those features are merged.
 
 
+## Proposing new clade designations
+
+Designating a new Nextstrain clade is a judgement call on top of a quantitative threshold:
+a candidate Pango lineage should carry **≥1 spike mutation** relative to its parent clade and
+have risen to **>30% regional** or **>20% global** frequency (see the
+[clade-naming blog post](https://nextstrain.org/blog/2022-04-29-SARS-CoV-2-clade-naming-2022)).
+We keep a strict 1:1 mapping between a Pango lineage and a Nextstrain clade name, and prefer to
+demarcate on the branch with the larger fitness differential.
+
+`scripts/propose_clades.py` automates the *evidence gathering* for this. It runs entirely on
+**open** (GenBank/INSDC) data — which is what we have since GISAID API access was lost (see the
+[2026-02-24 blog post](https://nextstrain.org/blog/2026-02-24-gisaid-joint-response)) and doesn't
+directly edit `defaults/`. It assembles a ranked candidate dossier; a human (or Claude Code, 
+see below) makes the call and writes the edits.
+
+### Running the script
+
+The script runs on **open** or **GISAID** data via `--data-source`, uses only the Python
+standard library, and fetches every input live (cached under `results/clade_cache/`, re-pulled
+with `--refresh`):
+
+```sh
+# Open (GenBank/INSDC): live global + North America + Europe core builds
+python3 scripts/propose_clades.py --data-source open
+
+# GISAID: auto-discovers your group's latest dated 6m builds (global + all 6 regions)
+python3 scripts/propose_clades.py --data-source gisaid --group blab
+```
+
+Both fetch the [Nextclade reference tree](https://nextstrain.org/nextclade/nextstrain/sars-cov-2/wuhan-hu-1/orfs)
+for clean lineage-defining mutations + parent clade, and the Nextstrain 6m builds (via charon
+`getDataset`) for region-filtered tip-frequencies. **open** also queries the
+[LAPIS open API](https://lapis.cov-spectrum.org/open/v2/) for an exact per-lineage read and the
+open [forecasts-ncov MLR](https://nextstrain.github.io/forecasts-ncov/) for fitness; **gisaid**
+uses the GISAID MLR endpoint and skips LAPIS (no API access). GISAID builds come from the group's
+Nextstrain Groups uploads (`groups/<group>/ncov/gisaid/<region>/6m/<date>`), auto-discovered via
+charon `getAvailable` — the latest upload date is chosen automatically (`--gisaid-date` pins an
+older one; `--gisaid-url-template` with `{region}` overrides discovery).
+
+Each geography's frequency is computed over **only the tips from that geography** (global = the
+global build unfiltered; each region = its build filtered to `region == <Region>`, the
+`?f_region=<Region>` equivalent), from the build's **subsample counts**. The build subsample is
+population-weighted (`group_by country/month` + `population_weights`), which is the object historical
+clade frequencies use and is deliberately *less* country-biased than raw GISAID counts (recent "Asia"
+in raw metadata is ~54% Singapore, which the build caps).
+
+Rather than read a single latest-pivot value (which rests on a handful of recent, KDE-extrapolated
+sequences), the threshold is judged as a **one-sided test on a proportion**: slide a **fixed-N=50**
+window of date-sorted tips across a backward-looking band (**~6 months ago → ~3 weeks ago**, the lag
+guard) and flag only if the **one-sided 95% Wilson lower bound** (rough Bonferroni over the
+effective independent windows, `m ≈ band/4wk`) clears the threshold (>20% global / >30% regional) in
+**some** window. Fixing the *sample size* (not the time-width) avoids splitting a real peak across
+under-powered windows; a flag therefore means *confidently* over the line, not a noisy point estimate
+(at N=50 the effective point-estimate bar is ≈46% regional / ≈34% global — `N` is a knob). A geography
+with **< 50** tips in the band reports `insufficient` and never flags. The dossier reports each flag's
+**peak window** (`peak X% [n, as of <date>, lo Y%]`) and the **current** window, so a lineage that was
+dominant months ago still surfaces, with its trajectory.
+
+Useful flags: `--window-n`, `--lookback-months`, `--lag-days`, `--refresh`, `--skip-lapis` /
+`--skip-mlr` / `--skip-regional`, and `--designation-year` (defaults to the current year — the name's
+year is the *designation* year, not emergence). Run `python3 scripts/propose_clades.py --help` for all.
+
+Outputs are written to (git-ignored) `results/`: `clade_candidates.md` (the human-readable dossier)
+and `clade_candidates.json` (the same data, structured, with paste-ready file edits).
+
+### Intended path for Claude Code
+
+This tool is designed to be driven by [Claude Code](https://claude.com/claude-code) via the
+`designate-clades` skill (`.claude/skills/designate-clades/SKILL.md`). The skill walks through the
+full loop:
+
+1. Run `scripts/propose_clades.py` and read `results/clade_candidates.md`.
+2. For each flagged candidate, cross-check against the live resources (the open builds at
+   `nextstrain.org/ncov/open/{global,north-america,europe}/6m`, the Cov-Spectrum lineage page,
+   forecasts-ncov, and the Nextclade reference tree).
+3. Decide the demarcation: among nested candidate lineages, elevate the one where the MLR growth
+   advantage jumps most and frequency is rising; or decline if it is just slicing the current
+   dominant clade finer.
+4. Write the edits (see below) and validate them.
+5. Draft the PR.
+
+Open the skill in a Claude Code session in this repo and ask it to find or designate new clades.
+
+### How a designation PR is made
+
+A designation edits five files in `defaults/` (the dossier prints a paste-ready block per
+candidate; trim the `clades.tsv` nuc rows to a characteristic handful — the spike-linked rows are
+flagged for you):
+
+1. `clades.tsv` — `<NAME>\tclade\t<PARENT>` plus a few `<NAME>\tnuc\t<site>\t<alt>` rows
+2. `clade_display_names.yml` — `<NAME>: <NAME> (<Pango>)`
+3. `clade_hierarchy.tsv` — `<NAME>\t<PARENT>\t<WHO>`
+4. `clade_emergence_dates.tsv` — `<NAME>\t<YYYY-MM-01>` (the lineage's first-seen date)
+5. `color_ordering.tsv` — `clade_membership\t<NAME> (<Pango>)`
+
+Validate that the new definition parses and inherits correctly through Augur (needs a Python
+environment with `augur` installed, e.g. the workflow's conda environment):
+
+```sh
+scripts/expand-clade-definitions defaults/clades.tsv | grep "^<NAME>"
+```
+
+Then open a pull request in the style of
+[#1149](https://github.com/nextstrain/ncov/pull/1149),
+[#1152](https://github.com/nextstrain/ncov/pull/1152),
+[#1158](https://github.com/nextstrain/ncov/pull/1158), and
+[#1192](https://github.com/nextstrain/ncov/pull/1192). The justification should state the new clade
+name and its Pango lineage, the parent clade, the defining spike mutations, the frequency evidence
+(with regions and numbers), the MLR growth advantage, and a one-line note on the demarcation
+choice. Keep the dossier's exact numbers in the PR body so reviewers can trace them.
+
 ## Running Core Nextstrain Builds
 
 The "core" nextstrain builds consist of a global analysis and six regional analyses, performed independently for GISAID data and open data (currently open data is GenBank data).
@@ -63,12 +176,12 @@ snakemake -pf upload deploy \
 ```
 
 
-## Triggering routine builds
+### Triggering routine builds
 
 Typically, everything’s triggered from the  `ncov-ingest` pipeline’s `trigger` command.
 After updating the intermediate files, that command will run the phylogenetic `ncov` pipelines (step 3, above) force-requiring the rules `deploy` and `upload`.
 
-## Triggering trial builds
+### Triggering trial builds
 
 This repository contains GitHub Actions `rebuild-gisaid` and `rebuild-open` which can be manually run [via github.com](https://github.com/nextstrain/ncov/actions).
 These will run the respective phylogenetic build pipelines starting from the preprocessed (filtered) files.
@@ -81,4 +194,4 @@ The GitHub Action `rebuild-100k` runs weekly (Mondays, via cron) and can also be
 It runs the `upload` target with `nextstrain_profiles/100k/config-open.yaml` to produce a roughly 100k-sequence open subsample — a smaller, representative dataset used for development and testing — and uploads the resulting intermediate files.
 See [remote inputs](https://docs.nextstrain.org/projects/ncov/en/latest/reference/remote_inputs.html) for where these files are published.
 
-If you want to test a particular branch, you can select the branch to use for the trial build when running the Github action.
+If you want to test a particular branch, you can select the branch to use for the trial build when running the Github action.