Skip to content

Commit 5eb310c

Browse files
fishidahoclaude
andauthored
Correct the held-out slice scale, and hold out half of each mode
PARAFAC2 writes slice k as `P_k (B diag(a_k)) C^T` with `P_k` orthonormal, so the reconstruction's norm is `||(B diag(a_k)) C^T||` regardless of how many cells the slice holds: all of a slice's energy sits in `a_k`. BiCV fit `a_k` against the training cells and then reused it unchanged for the held-out cells, and since a slice's energy grows like sqrt(n_k) that overstated the predicted block by sqrt(n_train_k / n_test_k) -- a factor of two at the old default split. The symptom was a metric that got worse as the model got better. On noiseless, exactly-rank-4 data fit at the true rank, the training block scored 0.9996 while the held-out block scored -0.034: the fit had recovered the structure exactly and the score still called it worse than predicting the mean. On the IBD cohort the curve was negative at every rank and fell monotonically as rank rose, putting its argmax at rank 1 and firing the "rank at the edge of tested ranks" warning. The error was purely one of magnitude -- a single global rescale of the prediction recovered R^2 = 0.970. `_holdout_scale` corrects each condition by sqrt(n_test_k / n_train_k), which needs only the cell counts and is exact when the two halves of a condition have comparable per-cell energy -- true here, since the split is random within each condition. Three candidates were measured (this one, re-fitting `a_k` per held-out slice from the train genes, and unconstrained least-squares cell loadings); all three fixed the bug and none scored above zero on pure noise, so the narrowest was taken. Both held-out fractions now default to one half, following Owen and Perry (https://arxiv.org/abs/0908.2062), who report that "in simulated examples we find that a method leaving out half the rows and half the columns performs well". That is also the one split at which the old score happened to be unbiased, so it is the setting under which older numbers remain comparable, and it recovers the true rank in every simulated scenario tested -- which the correction alone did not at 0.2. Measured after the change: noiseless rank-4 data at its own rank scores 0.97-0.99 at every fraction (was -0.034); the spread across fractions 0.1-0.7 falls from 3.75 to 0.02; argmax lands on the true rank for (4, noise 0.3), (6, noise 0.3) and (4, noise 1.0); pure noise still scores negative. Row-order invariance from the alignment fix underneath is undisturbed, agreeing to 8.9e-16 between contiguous and fully interleaved conditions. The dense reference in the tests derives the same factor independently from the counts, so it still pins the streaming algebra rather than the old behaviour. The tutorial's "peaks" claim is softened to describe the plateau the curve actually forms, since past the plateau neighbouring ranks differ only in the fourth decimal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016VJRB1RMUAb1id11XZ6KWS
1 parent 9c9108b commit 5eb310c

3 files changed

Lines changed: 174 additions & 11 deletions

File tree

‎docs/tutorial.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ plt.show()
9292

9393
### Select a Rank with Bi-Cross-Validation (BiCV)
9494

95-
$R^2X$ (in-sample fit) always increases with rank, so it cannot by itself identify an optimal rank — it can only reveal an elbow. Bi-cross-validation (BiCV) addresses this by holding out a random subset of cells *and* genes, fitting RISE on the remaining data, and scoring how well the fit predicts the held-out block. Because it is evaluated on unseen data, BiCV $R^2X$ penalizes overfitting and typically peaks (or plateaus) near the rank that generalizes best, unlike the fit $R^2X$.
95+
$R^2X$ (in-sample fit) always increases with rank, so it cannot by itself identify an optimal rank — it can only reveal an elbow. Bi-cross-validation (BiCV) addresses this by holding out a random subset of cells *and* genes, fitting RISE on the remaining data, and scoring how well the fit predicts the held-out block. Because it is evaluated on unseen data, BiCV $R^2X$ penalizes overfitting and turns over near the rank that generalizes best, unlike the fit $R^2X$. In practice the curve more often flattens into a plateau than forms a sharp peak, so read the elbow where it stops climbing rather than trusting the single highest point -- past the plateau, neighbouring ranks can differ only in the fourth decimal.
9696

9797
`scrise.rank_selection.bicv` exhaustively evaluates a list of candidate ranks, each with several repeated random train/test splits, and returns both the fit $R^2X$ and the BiCV $R^2X$ for every rank so you can inspect the full curve. Since PARAFAC2 fits use CANDELINC compression by default (`compress="auto"`), sweeping every rank in this way is cheap enough that there is no need to search for the best rank without evaluating every candidate.
9898

@@ -110,7 +110,7 @@ plt.tight_layout()
110110
plt.show()
111111
```
112112

113-
`bicv` returns a long-form DataFrame (columns `Rank`, `Repeat`, `Metric`, `R2X`, plus the per-trial diagnostics below) suitable for further analysis as well as plotting. Each BiCV trial holds out `held_out_cell_frac` of the cells within each condition and `held_out_gene_frac` of the genes (both default to 0.2); increase `n_repeats` for a smoother, less noisy BiCV curve at the cost of more compute.
113+
`bicv` returns a long-form DataFrame (columns `Rank`, `Repeat`, `Metric`, `R2X`, plus the per-trial diagnostics below) suitable for further analysis as well as plotting. Each BiCV trial holds out `held_out_cell_frac` of the cells within each condition and `held_out_gene_frac` of the genes. Both default to 0.5, the half-rows-and-half-columns split recommended by [Owen and Perry](https://arxiv.org/abs/0908.2062), who report that "in simulated examples we find that a method leaving out half the rows and half the columns performs well". Increase `n_repeats` for a smoother, less noisy BiCV curve at the cost of more compute.
114114

115115
Each BiCV row also carries the diagnostics needed to audit the curve rather than just plot it:
116116

‎scrise/rank_selection.py‎

Lines changed: 48 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,14 @@
88
predicts the held-out (test-cell x test-gene) block. Unlike the ordinary
99
in-sample fit R2X (which increases monotonically with rank), the BiCV R2X
1010
penalizes overfitting and typically peaks near the "true" rank of the data.
11+
12+
The held-out cells are scored through their own projections, whose scale has
13+
to be corrected for the size of the held-out slice -- see
14+
:func:`_holdout_scale`. Both held-out fractions default to one half, the
15+
split Owen and Perry recommend for bi-cross-validation
16+
(https://arxiv.org/abs/0908.2062); it is also the split at which the
17+
uncorrected score happened to be unbiased, so it is the setting under which
18+
older results are comparable to current ones.
1119
"""
1220

1321
import warnings
@@ -88,6 +96,34 @@ def _cell_loadings(
8896
return Z
8997

9098

99+
def _holdout_scale(
100+
cond_train: np.ndarray, cond_test: np.ndarray, n_cond: int
101+
) -> np.ndarray:
102+
"""Per-held-out-cell factor correcting `A` for the held-out slice's size.
103+
104+
PARAFAC2 writes slice ``k`` as ``P_k (B diag(a_k)) C^T`` with ``P_k``
105+
orthonormal, so the reconstruction's Frobenius norm is
106+
``||(B diag(a_k)) C^T||``, independent of how many cells the slice holds:
107+
all of a slice's energy sits in ``a_k``. ``a_k`` is fit against
108+
``n_train_k`` cells, and a slice's energy grows like ``sqrt(n_k)``, so
109+
reusing it unchanged for ``n_test_k`` held-out cells overstates the
110+
predicted block by ``sqrt(n_train_k / n_test_k)`` -- a factor of two at
111+
the default split, which drove the held-out R2X negative and made the
112+
curve fall as the fit improved.
113+
114+
Correcting it needs only the cell counts, which is exact when the two
115+
halves of a condition have comparable per-cell energy -- true here,
116+
since the split is random within each condition.
117+
"""
118+
scale = np.ones(cond_test.size)
119+
train_counts = np.bincount(cond_train, minlength=n_cond)
120+
test_counts = np.bincount(cond_test, minlength=n_cond)
121+
for i in range(n_cond):
122+
if test_counts[i] and train_counts[i]:
123+
scale[cond_test == i] = np.sqrt(test_counts[i] / train_counts[i])
124+
return scale[:, np.newaxis]
125+
126+
91127
# Nonzeros per row block when streaming column moments. Bounds the per-nonzero
92128
# temporaries to a few hundred MB regardless of how large the matrix is.
93129
_MOMENT_CHUNK_NNZ = 50_000_000
@@ -231,8 +267,10 @@ def _bicv_trial(
231267
cond_slices_test = condition_slices(cond_test, n_cond)
232268
P_test, _ = project_data(W_test, [A, B, C], cond_slices_test)
233269

234-
# Score the held-out block.
270+
# Score the held-out block. `A` carries the training slice's energy, so the
271+
# held-out loadings need rescaling for the held-out slice's size.
235272
L = _cell_loadings(P_test, B, A, cond_test, n_cond)
273+
L *= _holdout_scale(cond_train, cond_test, n_cond)
236274
LtY = np.asarray(
237275
rmatmul(np.ascontiguousarray(L.T), X_mat[test_cell_mask]), dtype=np.float64
238276
)[:, test_gene_idx]
@@ -266,8 +304,8 @@ def bicv(
266304
X: anndata.AnnData | None = None,
267305
ranks: Sequence[int] | None = None,
268306
n_repeats: int = 3,
269-
held_out_cell_frac: float = 0.2,
270-
held_out_gene_frac: float = 0.2,
307+
held_out_cell_frac: float = 0.5,
308+
held_out_gene_frac: float = 0.5,
271309
random_state: int | None = None,
272310
tolerance: float = 1e-6,
273311
max_iter: int = 200,
@@ -298,10 +336,14 @@ def bicv(
298336
n_repeats : int, optional (default: 3)
299337
Number of independent random cell/gene splits per rank. Higher
300338
values give a less noisy BiCV estimate but take longer.
301-
held_out_cell_frac : float, optional (default: 0.2)
339+
held_out_cell_frac : float, optional (default: 0.5)
302340
Fraction of cells held out per condition in each BiCV trial.
303-
held_out_gene_frac : float, optional (default: 0.2)
304-
Fraction of genes held out in each BiCV trial.
341+
held_out_gene_frac : float, optional (default: 0.5)
342+
Fraction of genes held out in each BiCV trial. Both default to a
343+
half-and-half split, following Owen and Perry, who report that
344+
"in simulated examples we find that a method leaving out half the
345+
rows and half the columns performs well"
346+
(https://arxiv.org/abs/0908.2062).
305347
random_state : int, optional
306348
Random seed for reproducibility.
307349
tolerance : float, optional (default: 1e-6)

‎scrise/tests/test_rank_selection.py‎

Lines changed: 124 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
from ..rank_selection import (
1414
_bicv_trial,
1515
_cell_loadings,
16+
_holdout_scale,
1617
_split_cells_by_condition,
1718
_split_genes,
1819
_test_block_moments,
@@ -325,8 +326,8 @@ def test_reported_seed_reproduces_its_own_trial():
325326
replayed = _bicv_trial(
326327
X,
327328
rank=3,
328-
held_out_cell_frac=0.2,
329-
held_out_gene_frac=0.2,
329+
held_out_cell_frac=0.5,
330+
held_out_gene_frac=0.5,
330331
seed=int(target["Seed"]),
331332
tolerance=1e-6,
332333
max_iter=60,
@@ -388,8 +389,12 @@ def dense(mat):
388389
sel = cond_test == i
389390
if not np.any(sel):
390391
continue
392+
n_train_i = int(np.sum(cond_train == i))
393+
# `A[i]` carries the training slice's energy; rescale it for this
394+
# slice's size (see `_holdout_scale`).
395+
scale = np.sqrt(int(sel.sum()) / n_train_i) if n_train_i else 1.0
391396
actual = X_test_test[sel]
392-
recon = ((P_test[i] @ B) * A[i]) @ C_test.T
397+
recon = (((P_test[i] @ B) * A[i]) * scale) @ C_test.T
393398
ss_res += float(np.sum((actual - recon) ** 2))
394399
ss_tot += float(np.sum(actual**2))
395400
return 1.0 - ss_res / ss_tot
@@ -500,3 +505,119 @@ def test_block_moments_stream_across_more_than_one_row_block(monkeypatch):
500505
block = dense[cell_mask]
501506
np.testing.assert_allclose(sums, block.sum(axis=0), rtol=1e-12)
502507
np.testing.assert_allclose(squares, np.sum(block**2, axis=0), rtol=1e-12)
508+
509+
510+
# Held-out slice scaling
511+
512+
513+
def _exact_low_rank(
514+
rank_true=4, noise=0.0, n_cond=6, cells=(90, 160), n_genes=90, seed=0
515+
):
516+
"""Data that IS exactly rank `rank_true`, with unequal cells per condition.
517+
518+
`means` is zero so the target stays exactly low rank -- mean-centering
519+
would add a rank-one term and put a ceiling below 1.0 on the fit.
520+
"""
521+
rng = np.random.default_rng(seed)
522+
C = rng.normal(size=(n_genes, rank_true))
523+
B = rng.normal(size=(rank_true, rank_true))
524+
A = np.abs(rng.normal(size=(n_cond, rank_true))) + 1.0
525+
blocks, cond = [], []
526+
for k in range(n_cond):
527+
n_k = int(rng.integers(*cells))
528+
P, _ = np.linalg.qr(rng.normal(size=(n_k, rank_true)))
529+
blocks.append(((P @ B) * A[k]) @ C.T)
530+
cond.append(np.full(n_k, k))
531+
mat = np.concatenate(blocks, axis=0)
532+
if noise:
533+
mat = mat + noise * np.std(mat) * rng.normal(size=mat.shape)
534+
adata = anndata.AnnData(X=mat)
535+
adata.obs["condition_unique_idxs"] = np.concatenate(cond)
536+
adata.var["means"] = np.zeros(n_genes)
537+
return adata
538+
539+
540+
def test_holdout_scale_is_the_square_root_of_the_cell_count_ratio():
541+
cond_train = np.repeat([0, 1, 2], [80, 40, 10])
542+
cond_test = np.repeat([0, 1, 2], [20, 40, 90])
543+
544+
scale = _holdout_scale(cond_train, cond_test, 3)
545+
546+
assert scale.shape == (cond_test.size, 1)
547+
np.testing.assert_allclose(scale[cond_test == 0, 0], np.sqrt(20 / 80))
548+
np.testing.assert_allclose(scale[cond_test == 1, 0], np.sqrt(40 / 40))
549+
np.testing.assert_allclose(scale[cond_test == 2, 0], np.sqrt(90 / 10))
550+
551+
552+
def test_holdout_scale_is_one_for_an_even_split():
553+
"""The split the defaults use, and the only one that needed no correction."""
554+
cond_train = np.repeat([0, 1], [50, 30])
555+
cond_test = np.repeat([0, 1], [50, 30])
556+
np.testing.assert_allclose(_holdout_scale(cond_train, cond_test, 2), 1.0)
557+
558+
559+
def test_holdout_scale_leaves_conditions_it_cannot_compare_alone():
560+
"""A condition absent from either side keeps a factor of one."""
561+
cond_train = np.array([0, 0, 0])
562+
cond_test = np.array([0, 2, 2])
563+
scale = _holdout_scale(cond_train, cond_test, 3)
564+
np.testing.assert_allclose(scale[cond_test == 2, 0], 1.0)
565+
566+
567+
def test_exactly_low_rank_data_scores_near_one_at_its_own_rank():
568+
"""The regression that matters: in-sample ~1 must not come with held-out < 0.
569+
570+
Before the slice-size correction this returned about -0.03 while the
571+
training block scored 0.9996 -- the model recovered the structure and the
572+
held-out score still said it was worse than predicting the mean.
573+
"""
574+
adata = _exact_low_rank(rank_true=4)
575+
trial = _bicv_trial(adata, 4, 0.5, 0.5, seed=0, tolerance=1e-8, max_iter=300)
576+
577+
assert trial["Train Block R2X"] > 0.99
578+
assert trial["BiCV R2X"] > 0.95
579+
580+
581+
@pytest.mark.parametrize("frac", [0.2, 0.35, 0.5, 0.7])
582+
def test_score_barely_moves_with_the_held_out_fraction(frac):
583+
"""`held_out_cell_frac` sets how much is held out, not what the score is.
584+
585+
The bug made this swing from -2.85 (frac 0.1) to 0.90 (frac 0.5) on the
586+
same data at the same rank, because the error was sqrt(n_train/n_test).
587+
"""
588+
adata = _exact_low_rank(rank_true=4)
589+
trial = _bicv_trial(adata, 4, frac, frac, seed=0, tolerance=1e-8, max_iter=300)
590+
assert trial["BiCV R2X"] > 0.9
591+
592+
593+
def test_bicv_recovers_a_known_rank():
594+
"""The point of the metric: peak (or plateau onset) at the true rank."""
595+
adata = _exact_low_rank(rank_true=3, noise=0.5, n_genes=60)
596+
results = bicv(adata, [1, 2, 3, 5, 8], n_repeats=2, random_state=0, max_iter=150)
597+
means = results[results["Metric"] == "BiCV R2X"].groupby("Rank")["R2X"].mean()
598+
599+
# Climbs up to the true rank, then stops climbing.
600+
assert means.loc[3] > means.loc[2] > means.loc[1]
601+
assert means.loc[5] <= means.loc[3] + 1e-3
602+
assert means.loc[8] <= means.loc[3] + 1e-3
603+
604+
605+
def test_pure_noise_does_not_score_positive():
606+
"""Nothing generalises from noise; a positive score would mean leakage."""
607+
rng = np.random.default_rng(0)
608+
cond = np.repeat(np.arange(5), 70)
609+
adata = anndata.AnnData(X=rng.normal(size=(cond.size, 60)))
610+
adata.obs["condition_unique_idxs"] = cond
611+
adata.var["means"] = np.zeros(60)
612+
613+
trial = _bicv_trial(adata, 5, 0.5, 0.5, seed=0, tolerance=1e-8, max_iter=200)
614+
assert trial["BiCV R2X"] < 0.02
615+
616+
617+
def test_held_out_fractions_default_to_one_half():
618+
"""Owen and Perry's recommended split; see the module docstring."""
619+
import inspect
620+
621+
params = inspect.signature(bicv).parameters
622+
assert params["held_out_cell_frac"].default == 0.5
623+
assert params["held_out_gene_frac"].default == 0.5

0 commit comments

Comments
 (0)