Stop pybids indexing the insides of image containers - #194
Closed
Arshya-Guru wants to merge 1 commit into
Closed
Conversation
pybids walks into directory-based image containers (.ims, .ome.zarr) and indexes every internal chunk file: on a 24-subject dataset that is ~645k of ~645k rows, costing ~8.5 GB RAM and 5+ minutes at every launch, which now exceeds the login node's 1.5 GB hard memory cap. Only the container path itself is a BIDS file, so prune descent into containers by overriding the BIDSLayoutIndexer constructor snakebids uses (it does not expose the indexer's ignore patterns). Dry-run peak memory drops from 8.5 GB to 381 MB and launch overhead from ~5.5 min to ~21 s, with a byte-identical job list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012437t8NSZLaZAwoAWK6bNT
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses excessive startup time and memory usage caused by pybids recursively indexing directory-based image containers (e.g., .ims, .ome.zarr) as if their internal chunk files were individual BIDS files. It mitigates this within the Snakemake entrypoint by overriding the indexer constructor that snakebids uses, pruning descent into such containers while preserving pybids’ default ignore locations.
Changes:
- Override
snakebids.core.input_generation.BIDSLayoutIndexerat Snakefile import time to inject additional ignore patterns for container directories. - Re-include pybids’
DEFAULT_LOCATIONS_TO_IGNOREso the override doesn’t drop the built-in ignore set.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+21
to
+26
| def _container_pruning_indexer(*args, **kwargs): | ||
| kwargs["ignore"] = list(_DEFAULT_IGNORE) + [ | ||
| _re.compile(r".*\.(ims|ome\.zarr|zarr)/") | ||
| ] | ||
| return _BIDSLayoutIndexer(*args, **kwargs) | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pybids walks into directory-based image containers (.ims, .ome.zarr) and indexes every internal chunk file: on a 24-subject dataset that is ~645k of ~645k rows, costing ~8.5 GB RAM and 5+ minutes at every launch, which now exceeds the login node's 1.5 GB hard memory cap. Only the container path itself is a BIDS file, so prune descent into containers by overriding the BIDSLayoutIndexer constructor snakebids uses (it does not expose the indexer's ignore patterns). Dry-run peak memory drops from 8.5 GB to 381 MB and launch overhead from ~5.5 min to ~21 s, with a byte-identical job list.