Skip to content

Commit 83ecb3b

Browse files
fishidahoclaude
andauthored
Correct the held-out slice scale, and hold out half of each mode (#552)
* Correct the held-out slice scale, and hold out half of each mode PARAFAC2 writes slice k as `P_k (B diag(a_k)) C^T` with `P_k` orthonormal, so the reconstruction's norm is `||(B diag(a_k)) C^T||` regardless of how many cells the slice holds: all of a slice's energy sits in `a_k`. BiCV fit `a_k` against the training cells and then reused it unchanged for the held-out cells, and since a slice's energy grows like sqrt(n_k) that overstated the predicted block by sqrt(n_train_k / n_test_k) -- a factor of two at the old default split. The symptom was a metric that got worse as the model got better. On noiseless, exactly-rank-4 data fit at the true rank, the training block scored 0.9996 while the held-out block scored -0.034: the fit had recovered the structure exactly and the score still called it worse than predicting the mean. On the IBD cohort the curve was negative at every rank and fell monotonically as rank rose, putting its argmax at rank 1 and firing the "rank at the edge of tested ranks" warning. The error was purely one of magnitude -- a single global rescale of the prediction recovered R^2 = 0.970. `_holdout_scale` corrects each condition by sqrt(n_test_k / n_train_k), which needs only the cell counts and is exact when the two halves of a condition have comparable per-cell energy -- true here, since the split is random within each condition. Three candidates were measured (this one, re-fitting `a_k` per held-out slice from the train genes, and unconstrained least-squares cell loadings); all three fixed the bug and none scored above zero on pure noise, so the narrowest was taken. Both held-out fractions now default to one half, following Owen and Perry (https://arxiv.org/abs/0908.2062), who report that "in simulated examples we find that a method leaving out half the rows and half the columns performs well". That is also the one split at which the old score happened to be unbiased, so it is the setting under which older numbers remain comparable, and it recovers the true rank in every simulated scenario tested -- which the correction alone did not at 0.2. Measured after the change: noiseless rank-4 data at its own rank scores 0.97-0.99 at every fraction (was -0.034); the spread across fractions 0.1-0.7 falls from 3.75 to 0.02; argmax lands on the true rank for (4, noise 0.3), (6, noise 0.3) and (4, noise 1.0); pure noise still scores negative. Row-order invariance from the alignment fix underneath is undisturbed, agreeing to 8.9e-16 between contiguous and fully interleaved conditions. The dense reference in the tests derives the same factor independently from the counts, so it still pins the streaming algebra rather than the old behaviour. The tutorial's "peaks" claim is softened to describe the plateau the curve actually forms, since past the plateau neighbouring ranks differ only in the fourth decimal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016VJRB1RMUAb1id11XZ6KWS * cleaing docs --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent efe0a7d commit 83ecb3b

3 files changed

Lines changed: 146 additions & 10 deletions

File tree

‎docs/tutorial.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ plt.show()
9292

9393
### Select a Rank with Bi-Cross-Validation (BiCV)
9494

95-
$R^2X$ (in-sample fit) always increases with rank, so it cannot by itself identify an optimal rank — it can only reveal an elbow. Bi-cross-validation (BiCV) addresses this by holding out a random subset of cells *and* genes, fitting RISE on the remaining data, and scoring how well the fit predicts the held-out block. Because it is evaluated on unseen data, BiCV $R^2X$ penalizes overfitting and typically peaks (or plateaus) near the rank that generalizes best, unlike the fit $R^2X$.
95+
$R^2X$ (in-sample fit) always increases with rank, so it cannot by itself identify an optimal rank. Bi-cross-validation (BiCV) addresses this by holding out a random subset of cells *and* genes, fitting RISE on the remaining data, and scoring how well the fit predicts the held-out block. Because it is evaluated on unseen data, BiCV $R^2X$ penalizes overfitting and turns over near the rank that generalizes best, unlike the fit $R^2X$.
9696

9797
`scrise.rank_selection.bicv` exhaustively evaluates a list of candidate ranks, each with several repeated random train/test splits, and returns both the fit $R^2X$ and the BiCV $R^2X$ for every rank so you can inspect the full curve. Since PARAFAC2 fits use CANDELINC compression by default (`compress="auto"`), sweeping every rank in this way is cheap enough that there is no need to search for the best rank without evaluating every candidate.
9898

@@ -110,7 +110,7 @@ plt.tight_layout()
110110
plt.show()
111111
```
112112

113-
`bicv` returns a long-form DataFrame (columns `Rank`, `Repeat`, `Metric`, `R2X`, plus the per-trial diagnostics below) suitable for further analysis as well as plotting. Each BiCV trial holds out `held_out_cell_frac` of the cells within each condition and `held_out_gene_frac` of the genes (both default to 0.2); increase `n_repeats` for a smoother, less noisy BiCV curve at the cost of more compute.
113+
`bicv` returns a long-form DataFrame (columns `Rank`, `Repeat`, `Metric`, `R2X`, plus the per-trial diagnostics below) suitable for further analysis as well as plotting. Each BiCV trial holds out `held_out_cell_frac` of the cells within each condition and `held_out_gene_frac` of the genes. Both default to 0.5, the half-rows-and-half-columns split recommended by [Owen and Perry](https://arxiv.org/abs/0908.2062), who report that "in simulated examples we find that a method leaving out half the rows and half the columns performs well". Increase `n_repeats` for a smoother, less noisy BiCV curve at the cost of more compute.
114114

115115
Each BiCV row also carries the diagnostics needed to audit the curve rather than just plot it:
116116

‎scrise/rank_selection.py‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,19 @@ def _cell_loadings(
8888
return Z
8989

9090

91+
def _holdout_scale(
92+
cond_train: np.ndarray, cond_test: np.ndarray, n_cond: int
93+
) -> np.ndarray:
94+
"""Per-held-out-cell factor correcting `A` for the held-out slice's size."""
95+
scale = np.ones(cond_test.size)
96+
train_counts = np.bincount(cond_train, minlength=n_cond)
97+
test_counts = np.bincount(cond_test, minlength=n_cond)
98+
for i in range(n_cond):
99+
if test_counts[i] and train_counts[i]:
100+
scale[cond_test == i] = np.sqrt(test_counts[i] / train_counts[i])
101+
return scale[:, np.newaxis]
102+
103+
91104
# Nonzeros per row block when streaming column moments. Bounds the per-nonzero
92105
# temporaries to a few hundred MB regardless of how large the matrix is.
93106
_MOMENT_CHUNK_NNZ = 50_000_000
@@ -231,8 +244,10 @@ def _bicv_trial(
231244
cond_slices_test = condition_slices(cond_test, n_cond)
232245
P_test, _ = project_data(W_test, [A, B, C], cond_slices_test)
233246

234-
# Score the held-out block.
247+
# Score the held-out block. `A` carries the training slice's energy, so the
248+
# held-out loadings need rescaling for the held-out slice's size.
235249
L = _cell_loadings(P_test, B, A, cond_test, n_cond)
250+
L *= _holdout_scale(cond_train, cond_test, n_cond)
236251
LtY = np.asarray(
237252
rmatmul(np.ascontiguousarray(L.T), X_mat[test_cell_mask]), dtype=np.float64
238253
)[:, test_gene_idx]
@@ -266,8 +281,8 @@ def bicv(
266281
X: anndata.AnnData | None = None,
267282
ranks: Sequence[int] | None = None,
268283
n_repeats: int = 3,
269-
held_out_cell_frac: float = 0.2,
270-
held_out_gene_frac: float = 0.2,
284+
held_out_cell_frac: float = 0.5,
285+
held_out_gene_frac: float = 0.5,
271286
random_state: int | None = None,
272287
tolerance: float = 1e-6,
273288
max_iter: int = 200,
@@ -298,9 +313,9 @@ def bicv(
298313
n_repeats : int, optional (default: 3)
299314
Number of independent random cell/gene splits per rank. Higher
300315
values give a less noisy BiCV estimate but take longer.
301-
held_out_cell_frac : float, optional (default: 0.2)
316+
held_out_cell_frac : float, optional (default: 0.5)
302317
Fraction of cells held out per condition in each BiCV trial.
303-
held_out_gene_frac : float, optional (default: 0.2)
318+
held_out_gene_frac : float, optional (default: 0.5)
304319
Fraction of genes held out in each BiCV trial.
305320
random_state : int, optional
306321
Random seed for reproducibility.

‎scrise/tests/test_rank_selection.py‎

Lines changed: 124 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
from ..rank_selection import (
1414
_bicv_trial,
1515
_cell_loadings,
16+
_holdout_scale,
1617
_split_cells_by_condition,
1718
_split_genes,
1819
_test_block_moments,
@@ -325,8 +326,8 @@ def test_reported_seed_reproduces_its_own_trial():
325326
replayed = _bicv_trial(
326327
X,
327328
rank=3,
328-
held_out_cell_frac=0.2,
329-
held_out_gene_frac=0.2,
329+
held_out_cell_frac=0.5,
330+
held_out_gene_frac=0.5,
330331
seed=int(target["Seed"]),
331332
tolerance=1e-6,
332333
max_iter=60,
@@ -388,8 +389,12 @@ def dense(mat):
388389
sel = cond_test == i
389390
if not np.any(sel):
390391
continue
392+
n_train_i = int(np.sum(cond_train == i))
393+
# `A[i]` carries the training slice's energy; rescale it for this
394+
# slice's size (see `_holdout_scale`).
395+
scale = np.sqrt(int(sel.sum()) / n_train_i) if n_train_i else 1.0
391396
actual = X_test_test[sel]
392-
recon = ((P_test[i] @ B) * A[i]) @ C_test.T
397+
recon = (((P_test[i] @ B) * A[i]) * scale) @ C_test.T
393398
ss_res += float(np.sum((actual - recon) ** 2))
394399
ss_tot += float(np.sum(actual**2))
395400
return 1.0 - ss_res / ss_tot
@@ -500,3 +505,119 @@ def test_block_moments_stream_across_more_than_one_row_block(monkeypatch):
500505
block = dense[cell_mask]
501506
np.testing.assert_allclose(sums, block.sum(axis=0), rtol=1e-12)
502507
np.testing.assert_allclose(squares, np.sum(block**2, axis=0), rtol=1e-12)
508+
509+
510+
# Held-out slice scaling
511+
512+
513+
def _exact_low_rank(
514+
rank_true=4, noise=0.0, n_cond=6, cells=(90, 160), n_genes=90, seed=0
515+
):
516+
"""Data that IS exactly rank `rank_true`, with unequal cells per condition.
517+
518+
`means` is zero so the target stays exactly low rank -- mean-centering
519+
would add a rank-one term and put a ceiling below 1.0 on the fit.
520+
"""
521+
rng = np.random.default_rng(seed)
522+
C = rng.normal(size=(n_genes, rank_true))
523+
B = rng.normal(size=(rank_true, rank_true))
524+
A = np.abs(rng.normal(size=(n_cond, rank_true))) + 1.0
525+
blocks, cond = [], []
526+
for k in range(n_cond):
527+
n_k = int(rng.integers(*cells))
528+
P, _ = np.linalg.qr(rng.normal(size=(n_k, rank_true)))
529+
blocks.append(((P @ B) * A[k]) @ C.T)
530+
cond.append(np.full(n_k, k))
531+
mat = np.concatenate(blocks, axis=0)
532+
if noise:
533+
mat = mat + noise * np.std(mat) * rng.normal(size=mat.shape)
534+
adata = anndata.AnnData(X=mat)
535+
adata.obs["condition_unique_idxs"] = np.concatenate(cond)
536+
adata.var["means"] = np.zeros(n_genes)
537+
return adata
538+
539+
540+
def test_holdout_scale_is_the_square_root_of_the_cell_count_ratio():
541+
cond_train = np.repeat([0, 1, 2], [80, 40, 10])
542+
cond_test = np.repeat([0, 1, 2], [20, 40, 90])
543+
544+
scale = _holdout_scale(cond_train, cond_test, 3)
545+
546+
assert scale.shape == (cond_test.size, 1)
547+
np.testing.assert_allclose(scale[cond_test == 0, 0], np.sqrt(20 / 80))
548+
np.testing.assert_allclose(scale[cond_test == 1, 0], np.sqrt(40 / 40))
549+
np.testing.assert_allclose(scale[cond_test == 2, 0], np.sqrt(90 / 10))
550+
551+
552+
def test_holdout_scale_is_one_for_an_even_split():
553+
"""The split the defaults use, and the only one that needed no correction."""
554+
cond_train = np.repeat([0, 1], [50, 30])
555+
cond_test = np.repeat([0, 1], [50, 30])
556+
np.testing.assert_allclose(_holdout_scale(cond_train, cond_test, 2), 1.0)
557+
558+
559+
def test_holdout_scale_leaves_conditions_it_cannot_compare_alone():
560+
"""A condition absent from either side keeps a factor of one."""
561+
cond_train = np.array([0, 0, 0])
562+
cond_test = np.array([0, 2, 2])
563+
scale = _holdout_scale(cond_train, cond_test, 3)
564+
np.testing.assert_allclose(scale[cond_test == 2, 0], 1.0)
565+
566+
567+
def test_exactly_low_rank_data_scores_near_one_at_its_own_rank():
568+
"""The regression that matters: in-sample ~1 must not come with held-out < 0.
569+
570+
Before the slice-size correction this returned about -0.03 while the
571+
training block scored 0.9996 -- the model recovered the structure and the
572+
held-out score still said it was worse than predicting the mean.
573+
"""
574+
adata = _exact_low_rank(rank_true=4)
575+
trial = _bicv_trial(adata, 4, 0.5, 0.5, seed=0, tolerance=1e-8, max_iter=300)
576+
577+
assert trial["Train Block R2X"] > 0.99
578+
assert trial["BiCV R2X"] > 0.95
579+
580+
581+
@pytest.mark.parametrize("frac", [0.2, 0.35, 0.5, 0.7])
582+
def test_score_barely_moves_with_the_held_out_fraction(frac):
583+
"""`held_out_cell_frac` sets how much is held out, not what the score is.
584+
585+
The bug made this swing from -2.85 (frac 0.1) to 0.90 (frac 0.5) on the
586+
same data at the same rank, because the error was sqrt(n_train/n_test).
587+
"""
588+
adata = _exact_low_rank(rank_true=4)
589+
trial = _bicv_trial(adata, 4, frac, frac, seed=0, tolerance=1e-8, max_iter=300)
590+
assert trial["BiCV R2X"] > 0.9
591+
592+
593+
def test_bicv_recovers_a_known_rank():
594+
"""The point of the metric: peak (or plateau onset) at the true rank."""
595+
adata = _exact_low_rank(rank_true=3, noise=0.5, n_genes=60)
596+
results = bicv(adata, [1, 2, 3, 5, 8], n_repeats=2, random_state=0, max_iter=150)
597+
means = results[results["Metric"] == "BiCV R2X"].groupby("Rank")["R2X"].mean()
598+
599+
# Climbs up to the true rank, then stops climbing.
600+
assert means.loc[3] > means.loc[2] > means.loc[1]
601+
assert means.loc[5] <= means.loc[3] + 1e-3
602+
assert means.loc[8] <= means.loc[3] + 1e-3
603+
604+
605+
def test_pure_noise_does_not_score_positive():
606+
"""Nothing generalises from noise; a positive score would mean leakage."""
607+
rng = np.random.default_rng(0)
608+
cond = np.repeat(np.arange(5), 70)
609+
adata = anndata.AnnData(X=rng.normal(size=(cond.size, 60)))
610+
adata.obs["condition_unique_idxs"] = cond
611+
adata.var["means"] = np.zeros(60)
612+
613+
trial = _bicv_trial(adata, 5, 0.5, 0.5, seed=0, tolerance=1e-8, max_iter=200)
614+
assert trial["BiCV R2X"] < 0.02
615+
616+
617+
def test_held_out_fractions_default_to_one_half():
618+
"""Owen and Perry's recommended split; see the module docstring."""
619+
import inspect
620+
621+
params = inspect.signature(bicv).parameters
622+
assert params["held_out_cell_frac"].default == 0.5
623+
assert params["held_out_gene_frac"].default == 0.5

0 commit comments

Comments
 (0)