Real fairness, calibration, and baseline-gap artifacts from the canonical run

README.md

@@ -43,7 +43,7 @@ Fusion vs. unimodal accuracy is **supporting evidence**, not the thesis.
 | **Cohort construction** | MIMIC-IV-Echo studies, MIMIC-IV-ECG records, MIMIC-IV demographics | Paired rows on `subject_id` within 24–48h, joined to structured LVEF, split by subject |
 | **Embedding extraction** | Paired cohort only (~few K–tens of K studies, not all 525K echos) | Frozen EchoJEPA-L (1024-d) and HuBERT-ECG (768-d) vectors, cached to Parquet |
 | **Probes** | Cached embeddings | ECG-only, echo-only (attentive), concat-MLP, and cross-attention fusion heads |
-| **Deployment analyses** | Trained fused checkpoint + held-out test split | Missing-modality degradation, fairness stratification, optional calibration |
+| **Deployment analyses** | Trained fused checkpoint + held-out test split | Missing-modality degradation, fairness stratification, EF≤40% calibration |
 
 Expensive forward passes run once and are cached; probe training reads only from cache and completes in minutes. Foundation-model weights are never fine-tuned.
 
@@ -53,26 +53,73 @@ See [TECHNICAL.md](./TECHNICAL.md) for full pipeline details.
 
 ## Results
 
-Held-out test split, n = 245. One fused cross-attention checkpoint (M09) trained on both modalities, scored under three inference-time conditions — no separate unimodal models are trained for the dropped conditions.
+Held-out test split, n = 245 after dropping three non-finite HuBERT-ECG rows from the original 248-row test split. One fused cross-attention checkpoint (M09) trained on both modalities, scored under three inference-time conditions — no separate unimodal models are trained for the dropped conditions.
 
-| Condition | Inference input | LVEF MAE | EF≤40% AUROC |
-|-----------|-----------------|---------:|-------------:|
-| `full` | Echo + ECG present | **10.28** | **0.766** |
-| `ecg_dropped` | ECG branch masked, echo present | 15.13 | 0.750 |
-| `echo_dropped` | Echo branch masked, ECG present | 18.57 | 0.693 |
+| Condition | Inference input | LVEF MAE (95% CI) | EF≤40% AUROC (95% CI) |
+|-----------|-----------------|-------------------:|----------------------:|
+| `full` | Echo + ECG present | **10.42** (9.28–11.64) | **0.771** (0.697–0.839) |
+| `ecg_dropped` | ECG branch masked, echo present | 11.22 (10.00–12.39) | 0.383 (0.279–0.493) |
+| `echo_dropped` | Echo branch masked, ECG present | 20.55 (19.02–22.02) | 0.689 (0.605–0.765) |
 
-Dropping echo costs more than dropping ECG on both metrics. AUROC holds up better than MAE under either drop, which is the graceful-degradation pattern rather than silent failure.
+Dropping echo costs much more MAE than dropping ECG, while ECG-dropped AUROC falls sharply. This run is a pooled-manifest baseline: the echo branch still receives one mean-pooled vector per study, not retained clip tokens — and on pooled inputs the cross-attention block provably cannot attend across modalities, so these numbers describe a concat-equivalent model (see [TECHNICAL.md §6.4](./TECHNICAL.md#6-probes)).
 
-**Provenance:** real run on cached EchoJEPA (`vjepa2.1-vitl-mimic-pt-100`) + HuBERT-ECG embeddings, point estimates transcribed from the E02 evaluation into `results/missing_modality.json`. Bootstrap confidence intervals need per-example predictions and are pending a canonical rerun. `results/` is gitignored, so that JSON is not in this repository — see [CONTRIBUTING.md](./CONTRIBUTING.md#reproducibility) for what reproducing these numbers takes.
+**Provenance:** real run on cached EchoJEPA (`vjepa2.1-vitl-mimic-pt-100`) + HuBERT-ECG embeddings, local Mac CPU, seed 42, fusion width 256, 1,000 bootstrap resamples. The canonical pooled fused checkpoint was selected by validation MAE from an all-probe M10 run (`full` val MAE 10.62). `results/` and `probes/` are gitignored, so checkpoints, per-example predictions, and figures are local artifacts — see [CONTRIBUTING.md](./CONTRIBUTING.md#reproducibility) for what reproducing these numbers takes.
 
 Reproduce with:
 
 ```bash
-python scripts/evaluate_missing_modality.py
+python scripts/evaluate_missing_modality.py \
+  --manifest data/processed/echo_hubert_manifest.parquet \
+  --checkpoint probes/cross_attn_fused/cross_attn_fused.pt \
+  --embed-dim 256 \
+  --echo-dim 1024 \
+  --ecg-dim 768
 ```
 
 Defaults assume the standard cohort/embedding/checkpoint layout; `--help` lists the paths. `--embed-dim`, `--echo-dim`, and `--ecg-dim` must match the checkpoint architecture or the `state_dict` load fails on shape.
 
+### In-cohort baselines
+
+All four probes on identical splits (seed 42, same 245-row test frame, `scripts/diagnose_baseline_gap.py`):
+
+| Probe | LVEF MAE (95% CI) | EF≤40% AUROC (95% CI) |
+|-------|-------------------:|----------------------:|
+| Cross-attention fused | **10.42** (9.28–11.64) | **0.771** (0.697–0.839) |
+| Concat-MLP | 10.93 (9.77–12.22) | 0.716 (0.634–0.795) |
+| Echo-only (attentive) | 11.09 (9.79–12.25) | 0.771 (0.698–0.845) |
+| ECG-only (ridge) | 11.60 (10.36–12.84) | 0.671 (0.582–0.756) |
+
+Fusion beats ECG-only on both metrics with paired-bootstrap significance (ΔMAE −1.17, CI −1.99 to −0.33; ΔAUROC +0.100, CI +0.03 to +0.17) and is never behind either solo probe — the gap to published single-modality baselines is a cohort and label-regime property, not a fusion failure ([TECHNICAL.md §8](./TECHNICAL.md#8-metrics)).
+
+### Fairness
+
+Test-split stratification of the fused checkpoint (`scripts/evaluate_fairness.py`, same run as above; strata under n=20 are flagged in the artifact and omitted here):
+
+| Stratum | n | MAE `full` | MAE `echo_dropped` | AUROC `full` |
+|---------|--:|-----------:|-------------------:|-------------:|
+| Sex F | 144 | 10.13 | 21.19 | 0.770 |
+| Sex M | 101 | 10.84 | 19.63 | 0.763 |
+| Age 40–54 | 37 | 8.93 | 20.44 | 0.742 |
+| Age 55–64 | 57 | 9.43 | 23.67 | 0.727 |
+| Age 65–74 | 57 | 13.10 | 21.71 | 0.752 |
+| Age 75–89 | 77 | 10.29 | 18.29 | 0.789 |
+| Race: Black | 43 | 6.62 | 21.41 | 0.973 |
+| Race: White | 175 | 11.53 | 20.13 | 0.761 |
+
+The sex gap is small in both conditions. The largest full-condition gaps are age 65–74 (13.10 vs 10.42 overall) and the Black–White MAE difference (6.62 vs 11.53). Note that MIMIC-IV records administrative gender and admission-reported race, both with known curation bias, and the remaining race/age strata are too small for stable estimates (flagged, not silently included, in `results/fairness/`).
+
+### Calibration
+
+EF≤40% Platt scaler fit on val predictions only, applied to test (`scripts/evaluate_calibration.py`), 10 bins:
+
+| Condition | ECE |
+|-----------|----:|
+| `full` | 0.072 |
+| `echo_dropped` | 0.043 |
+| `ecg_dropped` | 0.015 |
+
+Post-scaling ECE stays low even with a modality dropped — but read `ecg_dropped` together with its 0.383 AUROC: predictions collapse toward the training mean, so the probabilities are calibrated yet nearly uninformative. Calibration alone is not evidence the degraded model is usable.
+
 ---
 
 ## Models & data
@@ -129,8 +176,11 @@ framework is the main contribution and lives in
 [src/primed_ai/failure/](./src/primed_ai/failure/); the fusion pipeline described above is the
 substrate it is instantiated on.
 
-Still open: PhysioNet credentialing and reserved GPU/storage, both needed for the canonical
-real-data rerun.
+The canonical pooled rerun is done — probe training and every deployment analysis run on a
+laptop CPU in about 90 seconds total, no GPU (see [TECHNICAL.md §9](./TECHNICAL.md#9-compute-and-infrastructure)).
+Still open: rebuilding the manifest with clip-level echo tokens (needs the gated HF shards),
+token-level ECG re-extraction (#72), and the cohort-expansion decision (#62) — the first two
+gate real cross-modal attention, the last gates statistical power.
 
 ---
 

TECHNICAL.md

@@ -51,10 +51,19 @@ Build the paired cohort **before** running any embedding extraction. Do not proc
 
 For each echo study:
 
-1. Match on `subject_id` to the nearest ECG record within a **24–48 hour** temporal window (using MIMIC timestamps).
+1. Match on `subject_id` to the nearest ECG record within a symmetric **±24 hour** window
+   (`build_cohort.py --window-hours 24`, the canonical build; the built manifest's
+   max |delta_hours| is 23.98).
 2. Join to the structured **LVEF** label from MIMIC-IV-Echo.
 
-Expected cohort size: a few thousand to tens of thousands of paired rows (exact count depends on window strictness and label availability).
+Current synchronized manifest size is 1,208 paired rows. Without extracting more
+EchoJEPA studies, the hard ceiling is 6,617 echo studies from subjects who also have an
+ECG; larger cohorts require additional echo embedding coverage, not just a looser join.
+
+Window sensitivity, measured on the built manifest: tightening to ±12h keeps 767 of the
+1,208 rows and ±6h keeps 515. Widening beyond ±24h cannot be measured from the manifest —
+it needs a cohort-database rebuild (`scripts/run_cohort_sensitivity.py` wraps the
+24h/48h/admission comparison into one command for whoever has BigQuery access).
 
 ### 3.2 Train / test split
 
@@ -108,7 +117,10 @@ Two ECG paths exist in this repo, and they are not interchangeable:
 Both are 768-d after pooling, so the probes accept either. Anything downstream that says "ECG
 embedding" below refers to the HuBERT-ECG Parquet unless stated otherwise.
 
-**Open decision:** pooling strategy for ECG embeddings — mean vs. attentive pooling. Resolve with a quick empirical check early in probe development.
+The current HuBERT-ECG Parquet stores one pooled vector per record. Mean-vs-attentive
+ECG pooling is therefore not a runnable ablation yet: attentive pooling over tiled copies
+of that vector is mathematically identical to mean pooling. Token-level ECG needs a
+re-extraction pass before this decision can be revisited.
 
 ---
 
@@ -142,6 +154,20 @@ All probes are trained on cached embeddings. Recommended build order: quick conc
 
 All four predict continuous LVEF; the EF≤40% gate is derived from that output.
 
+**What the cross-attention block can actually attend to (#72).** With the current inputs
+the fusion is one-directional at best. The HuBERT-ECG side arrives as one pooled vector
+tiled into identical tokens, so the `echo_to_ecg` attention output is provably independent
+of the echo query — identical keys force uniform softmax weights, and identical values make
+the weighted sum equal that value. That direction reduces to a fixed linear image of the
+pooled ECG vector (pinned by `test_echo_to_ecg_attention_is_degenerate_on_tiled_ecg_tokens`).
+`ecg_to_echo` attends genuinely only when clip-level echo tokens are retained
+(`build_echo_study_embeddings --max-clips`); on the pooled manifest both directions
+degenerate and the fused probe is equivalent to a concat model with per-modality linear
+pre-maps. Reported pooled-manifest numbers should be read accordingly. Decision: keep the
+cross-attention architecture — it fuses for real once clip-level echo tokens are in the
+manifest — and treat token-level ECG (which needs a re-extraction pass from raw waveforms;
+the parquet stores pooled vectors only) as the gate for restoring the second direction.
+
 ### 6.5 Prediction targets
 
 | Target | Type | Use |
@@ -174,7 +200,11 @@ For each condition, report:
 
 Plot a **degradation curve** across conditions. The key question: when echo is unavailable at inference, does the model degrade gracefully or fail silently?
 
-**Implementation note:** at echo-dropped evaluation, zero out or mask the echo branch of the fused probe rather than retraining a separate ECG-only model. This tests the actual deployed fused model under missing input.
+**Implementation note:** at dropped-modality evaluation, mask the selected fused-probe
+branch at inference time (`mask_echo` / `mask_ecg`) rather than retraining a unimodal
+model. This tests the deployed fused checkpoint under missing input. Learned null tokens
+and branch dropout are deferred unless the missing-modality rerun shows masking is
+unstable.
 
 ### 7.2 Fairness audit
 
@@ -193,21 +223,40 @@ Report per-stratum:
 
 Flag known MIMIC gender-curation bias in documentation. This analysis is cheap and directly addresses the equity evaluation pillar.
 
-### 7.3 Calibration (optional stretch)
+### 7.3 Calibration
 
-If time permits, assess calibration of the EF≤40% binary gate (e.g., reliability diagram, expected calibration error). A deployable risk score should produce well-calibrated probabilities, not just high AUROC.
+Assess calibration of the EF≤40% binary gate (reliability diagram, expected calibration
+error). A deployable risk score should produce well-calibrated probabilities, not just high
+AUROC. The Platt scaler is fit on val predictions only and applied to test
+(`scripts/evaluate_calibration.py`); measured numbers are in the
+[README results](README.md#results). Caveat when reading dropped-condition ECE: a model
+whose predictions collapse toward the training mean can look well calibrated after Platt
+scaling while discriminating no better than chance — report ECE alongside AUROC, never
+alone.
 
 ---
 
 ## 8. Metrics
 
 | Metric | Target | Baseline references |
 |---|---|---|
-| LVEF MAE | Continuous regression | EchoJEPA: 5.97 MAE |
-| EF≤40% AUROC | Binary clinical gate | ECG-FM: 0.929 AUROC |
+| LVEF MAE | Continuous regression | EchoJEPA: 5.97 MAE (published, different cohort) · in-cohort echo-only probe: 11.09 |
+| EF≤40% AUROC | Binary clinical gate | ECG-FM: 0.929 AUROC (published, different cohort) · in-cohort ECG-only probe: 0.671 |
 | Missing-modality degradation | Δ MAE / Δ AUROC across conditions | No external baseline — this is the novel result |
 | Fairness gap | Δ MAE / Δ AUROC across demographic strata | No external baseline |
 
+The published solo numbers are **not** like-for-like with this cohort. On identical splits
+(seed 42, 245-row test frame, `scripts/diagnose_baseline_gap.py`): fused 10.42 MAE / 0.771
+AUROC vs echo-only 11.09 / 0.771, ECG-only 11.60 / 0.671, concat 10.93 / 0.716. Paired
+bootstrap deltas: fused beats ECG-only on both metrics (ΔMAE −1.17, 95% CI [−1.99, −0.33];
+ΔAUROC +0.100, CI [0.03, 0.17]) and is never behind either solo probe. The gap to the
+published baselines is therefore a property of the cohort and label regime, not a fusion
+failure — even the in-cohort echo-only probe (same encoder as the published 5.97) lands at
+11.09. Contributing factors, measured: 821 training rows after the non-finite drop (829 in
+the split); heterogeneous LVEF label sources (the `lvef_upper` fallback contributes 6 test
+rows at 43.0 MAE — 18 of its 24 cohort rows carry a physiologically implausible 100.0 —
+versus 5.2 MAE on `lvef_3d` rows); and the mean-pooled echo regime (§6.4).
+
 **Pre-flight check:** confirm EF≤40% prevalence in the paired cohort is high enough for stable AUROC estimation before locking results (`scripts/check_ef40_prevalence.py`).
 
 Measured missing-modality numbers are in the [README results table](README.md#results); reproducing them is covered in [CONTRIBUTING.md](CONTRIBUTING.md#reproducibility).
@@ -218,9 +267,9 @@ Measured missing-modality numbers are in the [README results table](README.md#re
 
 | Resource | Requirement |
 |---|---|
-| GPU | H200 or equivalent; reserve early on ORCD |
-| Storage | Sufficient for cached embedding tensors across full paired cohort |
-| Runtime | Subsetting to paired cohort keeps extraction under ~1 hour (vs. hours for full 525K echo corpus) |
+| GPU | Not needed for cached-vector probe training; needed only for future ECG token re-extraction or new EchoJEPA extraction |
+| Storage | Pooled manifest 14 MB, one four-probe checkpoint set 7 MB, full results tree ~1 MB — everything fits under `data/`, `probes/`, `results/` in the repo working copy; `--max-clips` echo manifests scale linearly with retained clips |
+| Runtime | All CPU, measured on an Apple M1 Pro (16 GB): four-probe M10 training 47.7s wall-clock, missing-modality eval with 1,000 bootstrap resamples 11.8s, fairness stratification 8.3s, per-condition calibration ~5s. A full train + evaluate cycle is about 90 seconds |
 | Reproducibility | Fixed random seeds; logged hyperparameters; versioned embedding cache |
 
 Probe training after caching is CPU/GPU-light and completes in minutes.
@@ -242,12 +291,17 @@ This work differs from EchoingECG on three axes: frozen embeddings (no fine-tuni
 
 ## 11. Open technical decisions
 
-- [ ] ECG pooling: mean vs. attentive — empirical check on validation set
-- [ ] Echo↔ECG pairing window: 24h vs. 48h — sensitivity analysis or fixed choice with justification
-- [ ] Multi-match resolution: when multiple ECGs fall within the window, take nearest timestamp
-- [ ] EF≤40% prevalence in paired cohort — confirm before reporting AUROC
-- [ ] GPU + storage budget for embedding cache on ORCD
-- [ ] Missing-modality masking strategy: zero-out vs. learned null token vs. branch dropout at eval only
+- [x] ECG pooling: mean — the parquet stores one pooled vector per record, and attentive
+      pooling over tiled copies of it is mathematically identical to mean pooling, so this
+      is the only runnable choice. Revisit only after a token-level ECG re-extraction (#72).
+- [x] Echo↔ECG pairing window: fixed at ±24h — this is what the canonical cohort was built
+      with (max |delta_hours| 23.98). Within-manifest tightening loses rows fast (767 at
+      ±12h, 515 at ±6h); widening to 48h needs a cohort-database rebuild and is scoped
+      under D07 (#62) via `scripts/run_cohort_sensitivity.py`.
+- [x] Multi-match resolution: when multiple ECGs fall within the window, take nearest timestamp
+- [x] EF≤40% prevalence in paired cohort: train/test are reportable; validation has only 26 EF<=40 positives, so validation AUROC should be treated as unstable
+- [x] GPU + storage budget for cached probe training: no GPU needed; current pooled manifest and checkpoints are laptop-scale
+- [x] Missing-modality masking strategy: use inference-time branch masks; no learned null token or branch dropout for the canonical rerun
 
 ---
 

configs/cohort/default.yaml

@@ -1,5 +1,8 @@
-pairing_window_hours: 48
+# Symmetric window in hours on EACH side of the echo (build_cohort.py --window-hours
+# semantics). The canonical 1,208-row manifest was built at 24, i.e. ECG within +/-24h
+# of the echo; the built manifest's max |delta_hours| is 23.98.
+pairing_window_hours: 24
 split_seed: 42
 train_frac: 0.70
 val_frac: 0.10
-test_frac: 0.20
+test_frac: 0.20

configs/encoder/ecg_fm.yaml

@@ -1,18 +1,25 @@
 name: ecg_fm
 repo_id: wanglab/ecg-fm
 checkpoint: mimic_iv_ecg_physionet_pretrained.pt
+# Intentionally a local override: ECG-FM weights are only needed for the alternative
+# extraction path, which has produced no reported number. Point at your own download.
 checkpoint_path: /path/to/shared/weights/mimic_iv_ecg_physionet_pretrained.pt
 embed_dim: 768
 seq_len: 64
 sample_rate: 500
 signal_length: 5000
 n_leads: 12
 frozen: true
+# ECG pooling decision (E11/#69): the HuBERT-ECG parquet stores one pooled vector per
+# record, so mean pooling is the only runnable option — attentive pooling over tiled
+# copies of a pooled vector is mathematically identical. Revisit only after a
+# token-level ECG re-extraction (#72).
 pooling: mean
 # Set use_stub: true for offline dev/CI without fairseq_signals + checkpoint.
 use_stub: false
 # Everything above configures the ECG-FM wrapper (encoders/ecg_fm.py), an alternative
 # extraction path that has not produced any reported result. The probes and evaluation
 # scripts read the pre-extracted HuBERT-ECG embeddings below — these are the ones behind
-# every reported ECG number.
-hubert_ecg_parquet: /path/to/shared/ecg-hubert/mimic-iv-ecg-ve.parquet
+# every reported ECG number. The repo-local canonical copy lands here via the team share;
+# see docs/embeddings.md §2 and docs/echo_hubert_results.md for how to obtain it.
+hubert_ecg_parquet: data/interim/hubert_ecg_embeddings.parquet