Skip to content

Commit c9d1daf

Browse files
aarmeyclaude
andauthored
Keep BiCV's held-out cell blocks lazy for duck-typed backends (#556)
* Keep BiCV's held-out cell blocks lazy for duck-typed backends _bicv_trial evaluated a rank's fit by row-restricting X.X with a boolean cell mask (X_mat[train_cell_mask]/X_mat[test_cell_mask]) and handing the result to rmatmul/calc_W, deliberately avoiding materializing the raw data ("reaches the raw data through products... rather than materialising a block of it"). That holds for a plain ndarray or scipy-sparse X_mat, but not for a vsparse normalized view (e.g. BAL-Pf2's lazy-normalized-view AnnData): bracket indexing such a view is documented to always eagerly build a dense ndarray for the selection. At BAL-Pf2's real scale (1.3M cells), a single train/test split (~50% of all cells) would materialize on the order of tens of GB, once or twice per BiCV trial, across every rank/repeat in a sweep -- never surfaced before because an unrelated vsparse memory issue always killed these runs earlier in the pipeline. Adds _restrict_rows(X_mat, mask), which uses vsparse's new select(recalculate=False) (meyer-lab/vsparse#50) to stay a genuinely lazy view for a duck-typed backend, falling back to ordinary indexing for plain dense/sparse X_mat (unaffected, still cheap for those). Also adds a third branch to _test_block_moments for the same duck-typed case: selects the cell subset lazily, then streams it in bounded row chunks rather than ever materializing the whole subset as one dense block. Temporarily pins vsparse to its (as yet unmerged/unreleased) select-without-recalculate branch in pyproject.toml -- drop once meyer-lab/vsparse#50 merges and releases, reverting to the plain PyPI version constraint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Point the temporary vsparse pin at the combined testing branch Avoids a conflicting-git-ref resolution error for downstream consumers (e.g. BAL-Pf2) that pin vsparse's combined bal-pf2-gpu-testing branch (carrying both #49's matmul kernel and #50's select(recalculate=False)) rather than #50's branch alone. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Switch vsparse dependency back to PyPI, pinned to >=0.4.0 The matmul kernel and lazy select(recalculate=False) this branch needed have been released, so drop the temporary git pin to the combined testing branch in favor of the PyPI release. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 5846d0c commit c9d1daf

4 files changed

Lines changed: 191 additions & 12 deletions

File tree

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ dependencies = [
1717
"anndata>=0.13",
1818
"pacmap>=0.9",
1919
"tqdm>=4.66.1",
20-
"vsparse>=0.2.0",
20+
"vsparse>=0.4.0",
2121
"hdf5plugin>=7.0.0",
2222
"matplotlib>=3.8",
2323
"seaborn>=0.13.2",

‎scrise/rank_selection.py‎

Lines changed: 53 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,31 @@ def _holdout_scale(
105105
# temporaries to a few hundred MB regardless of how large the matrix is.
106106
_MOMENT_CHUNK_NNZ = 50_000_000
107107

108+
# Row-chunk size (in bytes of the dense block materialized per chunk) when
109+
# streaming a duck-typed backend (e.g. a vsparse normalized view) that has no
110+
# CSR internals to stream directly, but does support a lazy, stats-preserving
111+
# `select()`. Keeps a chunk's dense materialization bounded regardless of how
112+
# large the selected block is.
113+
_MOMENT_CHUNK_BUDGET_BYTES = 64 << 20
114+
115+
116+
def _restrict_rows(X_mat: Any, mask: np.ndarray) -> Any:
117+
"""Row-restrict ``X_mat`` by a boolean mask, staying lazy where possible.
118+
119+
For a duck-typed backend that supports ``select()`` (e.g. a vsparse
120+
normalized view), keeps the view's *existing* statistics fixed rather
121+
than renormalizing the selected rows on their own -- correct here, since
122+
the caller is evaluating a model fit against a slice, not treating that
123+
slice as its own dataset -- and, unlike bracket indexing on such a view,
124+
never eagerly materializes the (potentially huge) selection as a dense
125+
array. Plain dense/scipy-sparse ``X_mat`` falls back to ordinary
126+
indexing, which is already cheap for those.
127+
"""
128+
select = getattr(X_mat, "select", None)
129+
if select is not None:
130+
return select(mask, recalculate=False)
131+
return X_mat[mask]
132+
108133

109134
def _test_block_moments(
110135
X_mat: Any, cell_mask: np.ndarray, gene_idx: np.ndarray
@@ -116,8 +141,8 @@ def _test_block_moments(
116141
sums = np.zeros(n_genes)
117142
squares = np.zeros(n_genes)
118143

119-
if not sps.issparse(X_mat):
120-
block = np.asarray(X_mat)[cell_mask]
144+
if isinstance(X_mat, np.ndarray):
145+
block = X_mat[cell_mask]
121146
# Both moments accumulate in float64. A float32 block summed in its own
122147
# dtype carries ~1e-4 relative error over a few hundred thousand rows,
123148
# which would reach the reported R2X through `ss_tot`.
@@ -126,6 +151,25 @@ def _test_block_moments(
126151
np.sum(block.astype(np.float64) ** 2, axis=0)[gene_idx],
127152
)
128153

154+
if not sps.issparse(X_mat):
155+
# A duck-typed backend (e.g. a vsparse normalized view): select the
156+
# cell subset lazily (keeping the view's existing statistics fixed,
157+
# rather than renormalizing this subset on its own -- see
158+
# `_restrict_rows`) and stream it in row chunks, rather than ever
159+
# materializing the whole (potentially huge) subset as one dense
160+
# block.
161+
sub = X_mat.select(cell_mask, recalculate=False)
162+
n_sub_rows = sub.shape[0]
163+
chunk_rows = max(1, _MOMENT_CHUNK_BUDGET_BYTES // (n_genes * 8))
164+
start = 0
165+
while start < n_sub_rows:
166+
stop = min(n_sub_rows, start + chunk_rows)
167+
block = np.asarray(sub[start:stop], dtype=np.float64)
168+
sums += block.sum(axis=0)
169+
squares += np.sum(block**2, axis=0)
170+
start = stop
171+
return sums[gene_idx], squares[gene_idx]
172+
129173
mat = X_mat.tocsr() if X_mat.format != "csr" else X_mat
130174
n_rows = mat.shape[0]
131175
start = 0
@@ -229,7 +273,8 @@ def _bicv_trial(
229273
cond_train = cond_idx[train_cell_mask]
230274
Z = _cell_loadings(P_train, B, A, cond_train, n_cond)
231275
ZtY = np.asarray(
232-
rmatmul(np.ascontiguousarray(Z.T), X_mat[train_cell_mask]), dtype=np.float64
276+
rmatmul(np.ascontiguousarray(Z.T), _restrict_rows(X_mat, train_cell_mask)),
277+
dtype=np.float64,
233278
)[:, test_gene_idx]
234279
ZtY -= np.outer(Z.sum(axis=0), means_test_genes)
235280
# `lstsq` on the rank x rank system keeps the minimum-norm
@@ -240,17 +285,18 @@ def _bicv_trial(
240285
C_full = np.zeros((n_genes, C.shape[1]))
241286
C_full[train_gene_mask] = C
242287
cond_test = cond_idx[test_cell_mask]
243-
W_test = calc_W(X_mat[test_cell_mask], means, C_full)
288+
X_test = _restrict_rows(X_mat, test_cell_mask)
289+
W_test = calc_W(X_test, means, C_full)
244290
cond_slices_test = condition_slices(cond_test, n_cond)
245291
P_test, _ = project_data(W_test, [A, B, C], cond_slices_test)
246292

247293
# Score the held-out block. `A` carries the training slice's energy, so the
248294
# held-out loadings need rescaling for the held-out slice's size.
249295
L = _cell_loadings(P_test, B, A, cond_test, n_cond)
250296
L *= _holdout_scale(cond_train, cond_test, n_cond)
251-
LtY = np.asarray(
252-
rmatmul(np.ascontiguousarray(L.T), X_mat[test_cell_mask]), dtype=np.float64
253-
)[:, test_gene_idx]
297+
LtY = np.asarray(rmatmul(np.ascontiguousarray(L.T), X_test), dtype=np.float64)[
298+
:, test_gene_idx
299+
]
254300
LtY -= np.outer(L.sum(axis=0), means_test_genes)
255301

256302
col_sums, col_squares = _test_block_moments(X_mat, test_cell_mask, test_gene_idx)

‎scrise/tests/test_rank_selection.py‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -507,6 +507,139 @@ def test_block_moments_stream_across_more_than_one_row_block(monkeypatch):
507507
np.testing.assert_allclose(squares, np.sum(block**2, axis=0), rtol=1e-12)
508508

509509

510+
# -- duck-typed backend: a vsparse normalized view ---------------------------
511+
#
512+
# `X.X` for a real vsparse-backed dataset (e.g. BAL-Pf2's lazy-normalized-view
513+
# AnnData) is neither a plain ndarray nor scipy-sparse -- it's a
514+
# VCSRArrayNormalized/VCSCArrayNormalized. Bracket indexing it with a large
515+
# boolean mask (the old `X_mat[cell_mask]`) eagerly materializes the whole
516+
# selection as dense, which is fine at test scale but would be tens of GB at
517+
# BAL-Pf2's real scale. `_restrict_rows`/`_test_block_moments`'s duck-typed
518+
# branch exist to avoid that; these tests exercise them directly.
519+
520+
521+
def _normalized_view(rng, shape=(120, 30), vsparse_cls=None):
522+
import vsparse
523+
524+
vsparse_cls = vsparse_cls or vsparse.VCSRArray
525+
dense = rng.random(shape)
526+
dense[dense < 0.6] = 0.0
527+
mat = (
528+
sps.csr_array(dense)
529+
if vsparse_cls is vsparse.VCSRArray
530+
else sps.csc_array(dense)
531+
)
532+
return dense, vsparse_cls.from_scipy(mat).normalized()
533+
534+
535+
@pytest.mark.parametrize("fmt", ["csr", "csc"])
536+
def test_restrict_rows_matches_bracket_indexing_for_a_normalized_view(fmt):
537+
import vsparse
538+
539+
rng = np.random.default_rng(2)
540+
vsparse_cls = vsparse.VCSRArray if fmt == "csr" else vsparse.VCSCArray
541+
_dense, nv = _normalized_view(rng, vsparse_cls=vsparse_cls)
542+
mask = rng.random(nv.shape[0]) < 0.6
543+
544+
from ..rank_selection import _restrict_rows
545+
546+
restricted = _restrict_rows(nv, mask)
547+
assert isinstance(restricted, type(nv)) # stayed a lazy view, not densified
548+
np.testing.assert_allclose(
549+
np.asarray(restricted.toarray()), np.asarray(nv[mask, :])
550+
)
551+
552+
553+
def test_restrict_rows_is_a_no_op_passthrough_for_plain_arrays():
554+
from ..rank_selection import _restrict_rows
555+
556+
rng = np.random.default_rng(3)
557+
dense = rng.random((20, 5))
558+
mask = rng.random(20) < 0.5
559+
560+
np.testing.assert_allclose(_restrict_rows(dense, mask), dense[mask])
561+
np.testing.assert_allclose(
562+
_restrict_rows(sps.csr_array(dense), mask).toarray(), dense[mask]
563+
)
564+
565+
566+
def test_block_moments_match_a_normalized_view_reference():
567+
"""`_test_block_moments`'s duck-typed branch matches a dense reference."""
568+
import vsparse
569+
570+
rng = np.random.default_rng(4)
571+
dense, nv = _normalized_view(rng, vsparse_cls=vsparse.VCSRArray)
572+
mask = rng.random(dense.shape[0]) < 0.7
573+
gene_idx = np.sort(rng.choice(dense.shape[1], size=11, replace=False))
574+
575+
sums, squares = _test_block_moments(nv, mask, gene_idx)
576+
block = np.asarray(nv[mask, :])[:, gene_idx]
577+
np.testing.assert_allclose(sums, block.sum(axis=0), rtol=1e-10)
578+
np.testing.assert_allclose(squares, np.sum(block**2, axis=0), rtol=1e-10)
579+
580+
581+
def test_block_moments_stream_a_normalized_view_across_more_than_one_chunk(monkeypatch):
582+
"""Force several row chunks so the normalized-view streaming path is exercised."""
583+
import vsparse
584+
585+
import scrise.rank_selection as rs
586+
587+
monkeypatch.setattr(rs, "_MOMENT_CHUNK_BUDGET_BYTES", 64) # forces 1-row chunks
588+
rng = np.random.default_rng(5)
589+
dense, nv = _normalized_view(rng, shape=(50, 12), vsparse_cls=vsparse.VCSRArray)
590+
mask = rng.random(dense.shape[0]) < 0.8
591+
gene_idx = np.arange(12)
592+
593+
sums, squares = rs._test_block_moments(nv, mask, gene_idx)
594+
block = np.asarray(nv[mask, :])
595+
np.testing.assert_allclose(sums, block.sum(axis=0), rtol=1e-10)
596+
np.testing.assert_allclose(squares, np.sum(block**2, axis=0), rtol=1e-10)
597+
598+
599+
def test_block_moments_streaming_peak_is_bounded_by_the_chunk_budget_not_the_block():
600+
"""The row-chunk streaming loop's own peak tracks the chunk size, not the
601+
selected block's total size.
602+
603+
``select(recalculate=False)`` on a boolean mask does its own one-time
604+
``O(nnz)`` structural rebuild (the same cost bracket indexing or any
605+
other full-array boolean selection pays in vsparse today) -- that part
606+
isn't what this streaming code is trying to bound, so it's built once,
607+
outside the measurement, isolating the chunk loop itself.
608+
"""
609+
import tracemalloc
610+
611+
import vsparse
612+
613+
rng = np.random.default_rng(6)
614+
dense, nv = _normalized_view(rng, shape=(4000, 400), vsparse_cls=vsparse.VCSRArray)
615+
mask = np.ones(dense.shape[0], dtype=bool) # select everything
616+
sub = nv.select(mask, recalculate=False)
617+
n_rows = sub.shape[0]
618+
619+
def _stream(chunk_rows: int) -> int:
620+
tracemalloc.start()
621+
try:
622+
tracemalloc.reset_peak()
623+
start = 0
624+
while start < n_rows:
625+
stop = min(n_rows, start + chunk_rows)
626+
block = np.asarray(sub[start:stop], dtype=np.float64)
627+
block.sum(axis=0)
628+
start = stop
629+
return tracemalloc.get_traced_memory()[1]
630+
finally:
631+
tracemalloc.stop()
632+
633+
_stream(50) # warm up any lazy imports/JIT before measuring
634+
small_chunk_peak = _stream(50)
635+
large_chunk_peak = _stream(n_rows) # the whole block in one "chunk"
636+
637+
# A peak that scaled with the full block regardless of chunk size would
638+
# make these roughly equal; bounded streaming keeps the small-chunk peak
639+
# well under the whole-block one.
640+
assert small_chunk_peak < large_chunk_peak / 4
641+
642+
510643
# Held-out slice scaling
511644

512645

‎uv.lock‎

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)