Skip to content

Cache Thomson PARAFAC2 factors and support on-demand IVCSR raw data download - #514

Merged
aarmey merged 5 commits into
mainfrom
cache-thomson-factors
Aug 27, 2026
Merged

aarmey merged 5 commits into
mainfrom
cache-thomson-factors

Conversation

@aarmey

@aarmey aarmey commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary of Changes

  1. Compact Cached Factors Dataset (Thomson_cached_factors.h5ad):

    • Built and added analysis/data/Thomson/Thomson_cached_factors.h5ad (~4.23 MB gzip compressed, down from 137 MB originally).
    • Removed X data and weighted_projections.
    • Downsampled projections to float16, factor matrices (Pf2_C, Pf2_A, Pf2_B, Pf2_weights) to float32, and condition_unique_idxs to int8.
    • Removed redundant constant doublet column.
  2. On-Demand Raw Data Download & IVCSR Support via vcsc:

    • Implemented download_thomson_raw() in analysis/imports.py to download the raw Thomson IVCSR-compressed dataset from Box (https://ucla.box.com/s/jy53rcort51xn5t2dr5927wfj13g9e7j) to analysis/data/Thomson/thomson_raw.h5ad when raw count data is required.
    • Updated import_thomson(), import_thomson_factors(include_raw=True), and Thomson_Doublet() to read the raw IVCSR-encoded dataset using vcsc.VCSCAnnData.
    • Added thomson_raw.h5ad to .gitignore.
  3. Figure Files Updated:

    • Updated figure generation scripts (figure2b_e.py, figure2f_h.py, figure3.py, figureS2.py, figureS3.py, figureS5.py, figureS6.py) to load data through import_thomson_factors().
  4. Testing:

    • Added unit test test_import_thomson_factors in analysis/tests/test_import.py.

aarmey added 5 commits August 24, 2026 06:01
… data

- Add export_factors and load_factors to RISE with OPQ projection compression (default R^2 >= 0.99)
- Store factor matrices (Pf2_A, Pf2_B, Pf2_weights, Pf2_C) in float32 and omit uncompressed/weighted projections
- Optionally store PaCMAP embeddings and rebuild full datasets on load via raw_path matching
- Convert Thomson_cached_factors.h5ad to OPQ representation
- Add comprehensive unit tests for OPQ, export_factors, and load_factors
…is/data/

- Remove analysis package from tool.uv.build-backend in pyproject.toml
- Ensure docs have zero dependencies on analysis
- Move Thomson_cached_factors.h5ad directly to analysis/data/
- Add export_factors and load_factors documentation to docs/api.rst
@aarmey
aarmey merged commit e193596 into main Aug 27, 2026
1 check passed
@aarmey
aarmey deleted the cache-thomson-factors branch August 27, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant