Fix NIDM append discarding all but one stats file; align with PyNIDM 4.5.0 + per-subject NIDM input

[yibeichan]

Summary

Aligns the FreeSurfer-NIDM BIDSapp with the PyNIDM 4.5.0 stack and the per-subject NIDM datasets now produced upstream (e.g. derivatives/nidm_4.5.0), so the app runs cleanly through BABS on the current study layout.

Validating that work surfaced a silent data-loss bug in the NIDM append path, fixed here: every NIDM file this app produced carried one stats file instead of 21. No such output has been merged into a shared dataset — the only run is still unmerged in a BABS output RIA — so nothing in circulation needs regenerating.

---

⚠️ Data-loss fix: all but one stats file was discarded

The append path passed -j to fs_to_nidm alongside -n. In fs_to_nidm's append branch, the per-stats-file loop accumulates by re-reading the -n file each iteration and serializing back to it. With -j the result is written to <nidm_file>.json instead — so every iteration restarts from the pristine -n file, and only the last-globbed stats file survives.

recon-all produces 21 stats files. One reached NIDM.

On ABIDE Caltech sub-0051456 the output carried 135 measurements from a single source (rh.BA_exvivo.thresh.stats). Absent: aseg.stats — all subcortical volumes, hippocampus/amygdala/thalamus/ventricles — and every aparc parcellation. Because glob order is filesystem-dependent, which file survived also varied by node: a local repro of the identical command kept mri_segstats instead, so the output was nondeterministic.

Verified against PyNIDM 4.5.0 + rdflib 7.6.0 on the real subject:

	stats sources	FSStatsCollections	measurements
-j (as shipped)	1	1	135
no -j (fixed)	19	19	5,188

The merge itself was always sound and remains so: the input NIDM is fully preserved (921/921 ground triples), and the FreeSurfer activity prov:used the original bidsmri2nidm T1w AcquisitionObject and is attributed to the same subject prov:Person.

This was invisible in job logs. The "files after conversion" listing enumerated the directory before the aggregated TTL was written, so the actual deliverable never appeared. It is now relabelled as intermediate output, and the aggregated TTL is logged with its size and triple count.

Shared CDE vocabulary no longer merged per subject

fs_cde.ttl is required to interpret the fs_* predicates — <collection> fs_000003 1664279.0 has zero describing triples without it. But it is static and byte-identical for every subject and run (md5 a36c6e07… across four independent outputs), and is already shipped as its own file in nidm/. Embedding it made every per-subject TTL unique and ~2.0 MB larger for no information gain.

All 5,188 measurements remain resolvable against the shared file — zero information lost. Per-subject TTL: 2,101,827 → 249,724 bytes. Because the file is byte-identical, git-annex now stores the vocabulary once per study rather than once per subject.

This matches the sibling fsl-nidm BIDSapp, which likewise ships fsl_cde.ttl alongside rather than inside its merged nidm.ttl.

Consumer note: downstream code must load nidm/fs_cde.ttl to resolve fs_* terms. This is a change from what this app previously shipped.

---

Other changes

• requirements.txt: pynidm 4.2.4 → 4.5.0; add explicit oxrdflib~=0.5.0 (oxigraph-backed rdflib store that PyNIDM 4.5.0 pulls in) and prov[graph]. prov 3.0.0 moved NetworkX graph interop to an optional graph extra that nidm.experiment imports — without it, NIDM conversion imports fail.
• Dockerfile / Singularity: cap the forced rdflib upgrade at >=7.0.0,<8 (oxrdflib requires rdflib<8).
• src/run.py: look for per-subject NIDM input (sub-<label>/nidm.ttl, session variants first) ahead of the legacy top-level nidm.ttl fallback. Backward compatible with single-file NIDM inputs; an absent subject yields no false match on another subject's file.
• src/segstats_jsonld (submodule): drop the direct rdflib-jsonld declaration — deprecated, JSON-LD moved into rdflib in 6.0. It still arrives transitively (pynidm → funowl → rdflib-shim → rdflib-jsonld==0.6.1), but rdflib-shim exists precisely to keep it inert under rdflib ≥6, so this is hygiene and does not change the installed set. Also synced the stale pynidm==4.2.4 pin, which was inert (the build never reads that file) but misleading.

Context

• The per-subject NIDM layout + PyNIDM 4.5.0 are the direction set in coordination with the NIDM dataset owner.
• Requires a BABS revision containing Stage BIDS-inherited metadata for every job; common_paths for explicit extras PennLINC/babs#385 (removes the hardcoded per-input dataset_description.json fetch; auto-resolves BIDS-inherited metadata). With older BABS the job fails at input setup on inputs lacking a top-level dataset_description.json.

Validation

• 4.5.0 SIF builds; in-container versions verified (pynidm 4.5.0, rdflib 7.6.0, oxrdflib 0.5.0, pyoxigraph 0.5.9, networkx 3.4.2) and the converter (segstats_jsonld.fs_to_nidm + nidm.experiment.*) imports resolve.
• Per-subject NIDM selection unit-tested (per-subject / legacy / session / absent-subject cases).
• A Caltech BABS run against nidm_4.5.0 (babs main) completed both subjects (19475545_1/2, exit 0:0) and pushed results to the output RIA.
• The append bug was found by parsing that run's output, reproduced against PyNIDM 4.5.0 with the real subject's stats directory, and fixed. The -o (no existing NIDM) branch was checked and is not affected — it writes one output per stats file, so nothing is overwritten.
• 79 tests pass in-container. Two regression tests added, each confirmed to fail with its fix reverted:
  • test_append_command_does_not_pass_jsonld
  • test_shared_cde_not_merged_into_per_subject_ttl

⏳ Still pending before merge

The fix is verified by direct fs_to_nidm reproduction, not yet through a rebuilt container. Remaining:

1. Rebuild the 4.5.0 SIF with these commits.
2. Re-run Caltech through BABS.
3. Confirm the merged TTL carries all 19 stats sources (expect ~5,188 measurements, with aseg subcortical volumes present).

Open question

Output layout still follows CLAUDE.md / ants_bidsapp (nidm/sub-<id>.ttl) rather than fsl-nidm's per-subject nidm.ttl + nidm_orig.ttl. The two sibling apps disagree; worth settling for cross-app interop, but out of scope here.

🤖 Generated with Claude Code

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.