Task metadata
| Field |
Value |
| Category |
Writing |
| Priority |
Medium |
| Complexity |
Low |
| Est. days |
1 |
| Related |
#68 (W07), PR #74 |
Description
The case-report PDF under paper/build/ uses failure-framework figures (taxonomy, complementarity) whose caption says "representative seed" — but the only artifacts that existed when it was built came from scripts/run_failure_analysis.py --demo, i.e. planted synthetic data (n=150). Nothing in the caption says synthetic, so the figures read as real-cohort results.
Since PR #74 the harness has a --manifest route and real artifacts exist:
python scripts/run_failure_analysis.py \
--manifest data/processed/echo_hubert_manifest.parquet --out results/failure
Real numbers on the 245-row test frame (ridge harness, seed-fixed): echo wins per-example 245/0, LOO marginal value echo 107.3 vs ecg 6.5, drop_echo induces 188 critical failures at silent rate 1.0, drop_ecg 59 at 0.27. The real story is stronger than the synthetic one — dropping echo fails silently across the board.
The .tex source is not in this repo (only paper/build/ outputs), so this needs whoever holds the paper source.
Steps:
- Regenerate the taxonomy / complementarity / dropout figures from
results/failure (real) and swap them into the paper.
- Update captions with real n and provenance; anywhere synthetic demo output remains, label it as harness validation on synthetic data.
- Rebuild the PDF so
paper/build/ stops carrying figures over unlabeled synthetic numbers.
Acceptance criteria:
Task metadata
Description
The case-report PDF under
paper/build/uses failure-framework figures (taxonomy, complementarity) whose caption says "representative seed" — but the only artifacts that existed when it was built came fromscripts/run_failure_analysis.py --demo, i.e. planted synthetic data (n=150). Nothing in the caption says synthetic, so the figures read as real-cohort results.Since PR #74 the harness has a
--manifestroute and real artifacts exist:Real numbers on the 245-row test frame (ridge harness, seed-fixed): echo wins per-example 245/0, LOO marginal value echo 107.3 vs ecg 6.5, drop_echo induces 188 critical failures at silent rate 1.0, drop_ecg 59 at 0.27. The real story is stronger than the synthetic one — dropping echo fails silently across the board.
The
.texsource is not in this repo (onlypaper/build/outputs), so this needs whoever holds the paper source.Steps:
results/failure(real) and swap them into the paper.paper/build/stops carrying figures over unlabeled synthetic numbers.Acceptance criteria: