Skip to content

Commit 22ce5a8

Browse files
fishidahoclaude
andcommitted
Report the per-trial diagnostics a rank sweep needs
Closes #548. `bicv` returned `Rank / Repeat / Metric / R2X`, which is enough to plot a curve but not enough to audit the rank it implies. BiCV rows now also carry: Train Block R2X in-sample R2X on the block the model was fit to NTrainGenes, NTestGenes, NTrainCells, NTestCells Seed the seed determining that trial's splits and init `Train Block R2X` is the one that changes what the output can be used for. The useful reading of a BiCV curve is the held-out score turning over while the in-sample score keeps climbing, and the existing "Fit R2X" metric cannot supply that comparison: it is a separate fit, on the full dataset rather than on the training block, so it answers a different question. The value was already being computed and discarded. `_bicv_trial` now takes an explicit `seed` and builds its own generator instead of drawing from a shared one, which is what makes `Seed` meaningful -- a single trial can be replayed from the value in the table. A test asserts that replay reproduces the recorded scores exactly. The block sizes are reported rather than assumed because the cell split is stratified per condition, so the realised fraction is only approximately the requested one, and the spread across repeats has to be read against the block each repeat actually used. These columns are NaN on "Fit R2X" rows, which come from an unsplit fit. The additions are columns rather than a new `Metric` value, so `plot_bicv_r2x` -- which colours by `Metric` -- is unchanged. The `test_bicv_shape_and_range` column assertion is relaxed from an exact set to a subset, since the return shape is intentionally wider. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 0de3096 commit 22ce5a8

3 files changed

Lines changed: 170 additions & 12 deletions

File tree

‎docs/tutorial.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,24 @@ plt.tight_layout()
110110
plt.show()
111111
```
112112

113+
Each BiCV row also carries the diagnostics needed to audit the curve rather than
114+
just plot it:
115+
116+
| column | what it is for |
117+
| :----- | :------------- |
118+
| `Train Block R2X` | In-sample $R^2X$ on the block the model was fit to. The reading that matters is this climbing while the held-out $R^2X$ turns over — the separate `Fit R2X` metric is a different fit on different data and cannot play that role. |
119+
| `NTrainGenes`, `NTestGenes`, `NTrainCells`, `NTestCells` | The realised block sizes, which set the scale of the spread across repeats and confirm the split is what you asked for. |
120+
| `Seed` | The seed determining that trial's splits and initialisation, so a single trial can be rerun in isolation. |
121+
122+
```python
123+
# Does the held-out score turn over while the in-sample one keeps climbing?
124+
trials = results[results["Metric"] == "BiCV R2X"]
125+
print(trials.groupby("Rank")[["R2X", "Train Block R2X"]].mean())
126+
```
127+
128+
These columns are `NaN` on the `Fit R2X` rows, which come from an unsplit fit on
129+
the full dataset.
130+
113131
`bicv` returns a long-form DataFrame (columns `Rank`, `Repeat`, `Metric`, `R2X`) suitable for further analysis as well as plotting. Each BiCV trial holds out `held_out_cell_frac` of the cells within each condition and `held_out_gene_frac` of the genes (both default to 0.2); increase `n_repeats` for a smoother, less noisy BiCV curve at the cost of more compute.
114132

115133
If you plan to fit the final model with non-default `parafac2_nd` options (e.g. `normalize_slices=True`, see [Handling Unequal Cell Counts](unequal_cell_counts.md)), pass the same options to `bicv` via `parafac2_kwarg` so that rank selection is evaluated under the same fitting behavior:

‎scrise/rank_selection.py‎

Lines changed: 55 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -106,14 +106,14 @@ def _bicv_trial(
106106
rank: int,
107107
held_out_cell_frac: float,
108108
held_out_gene_frac: float,
109-
rng: np.random.Generator,
109+
seed: int,
110110
tolerance: float,
111111
max_iter: int,
112112
compress: int | tuple[int, int | None] | str | bool | None = "auto",
113113
compression_kwarg: dict[str, Any] | None = None,
114114
parafac2_kwarg: dict[str, Any] | None = None,
115-
) -> float:
116-
"""Run a single bi-cross-validation trial and return the held-out R2X.
115+
) -> dict[str, float]:
116+
"""Run a single bi-cross-validation trial and return its scores and shape.
117117
118118
Splits cells (stratified by condition) and genes into train/test blocks,
119119
fits PARAFAC2 on the train-cell x train-gene block, then predicts the
@@ -128,7 +128,18 @@ def _bicv_trial(
128128
129129
R2X is then computed by reconstructing the held-out block from these
130130
estimates and comparing against the (mean-centered) observed values.
131+
132+
``seed`` fully determines the trial: the gene split, the cell split and the
133+
fit's initialisation all derive from it, so a single trial can be rerun or
134+
re-scored in isolation from the value reported in the results table.
135+
136+
Returns
137+
-------
138+
dict[str, float]
139+
``BiCV R2X`` (the held-out block), ``Train Block R2X`` (in-sample, on
140+
the block the model was fit to), the four block sizes, and ``Seed``.
131141
"""
142+
rng = np.random.default_rng(seed)
132143
cond_idx = X.obs["condition_unique_idxs"].to_numpy().astype(int)
133144
n_cond = int(cond_idx.max()) + 1
134145
means = X.var["means"].to_numpy() if "means" in X.var else np.zeros(X.n_vars)
@@ -138,7 +149,7 @@ def _bicv_trial(
138149
train_gene_mask, test_gene_mask = _split_genes(X.n_vars, held_out_gene_frac, rng)
139150

140151
X_train = X[train_cell_mask][:, train_gene_mask].copy()
141-
(weights, (A, B, C), P_train), _ = run_parafac2(
152+
(weights, (A, B, C), P_train), train_block_r2x = run_parafac2(
142153
X_train,
143154
rank=rank,
144155
random_state=int(rng.integers(np.iinfo(np.int32).max)),
@@ -181,7 +192,15 @@ def _bicv_trial(
181192
ss_res += float(np.sum((actual - recon) ** 2))
182193
ss_tot += float(np.sum(actual**2))
183194

184-
return 1.0 - ss_res / ss_tot
195+
return {
196+
"BiCV R2X": 1.0 - ss_res / ss_tot,
197+
"Train Block R2X": float(train_block_r2x),
198+
"NTrainGenes": int(train_gene_mask.sum()),
199+
"NTestGenes": int(test_gene_mask.sum()),
200+
"NTrainCells": int(train_cell_mask.sum()),
201+
"NTestCells": int(test_cell_mask.sum()),
202+
"Seed": int(seed),
203+
}
185204

186205

187206
def bicv(
@@ -250,9 +269,26 @@ def bicv(
250269
Returns
251270
-------
252271
pandas.DataFrame
253-
Long-form DataFrame with columns "Rank", "Repeat", "Metric"
254-
(one of "Fit R2X" or "BiCV R2X"), and "R2X". Ready to pass to
272+
Long-form DataFrame with columns "Rank", "Repeat", "Metric" (one of
273+
"Fit R2X" or "BiCV R2X"), and "R2X". Ready to pass to
255274
:func:`scrise.plotting.plot_bicv_r2x`.
275+
276+
BiCV rows carry per-trial diagnostics as additional columns:
277+
278+
``Train Block R2X``
279+
In-sample R2X on the block the model was actually fit to. The
280+
useful reading of a BiCV curve is that this keeps climbing while
281+
the held-out R2X turns over; the separate "Fit R2X" metric is a
282+
different fit on different data and cannot play that role.
283+
``NTrainGenes``, ``NTestGenes``, ``NTrainCells``, ``NTestCells``
284+
The realised block sizes, which set the scale of the spread across
285+
repeats and confirm the split matches what was requested.
286+
``Seed``
287+
The seed that determines the trial's splits and initialisation, so
288+
a single trial can be rerun in isolation.
289+
290+
These columns are NaN on "Fit R2X" rows, which come from an
291+
unsplit fit on the full dataset.
256292
"""
257293
if X is None and adata is not None:
258294
X = adata
@@ -301,23 +337,32 @@ def bicv(
301337
compression_kwarg=compression_kwarg,
302338
parafac2_kwarg=parafac2_kwarg,
303339
)
340+
# The full-data fit has no split, so the per-trial columns are absent
341+
# here rather than zero; pandas fills them with NaN.
304342
rows.append({"Rank": rank, "Repeat": 0, "Metric": "Fit R2X", "R2X": fit_r2x})
305343

306344
for repeat in range(n_repeats):
307-
bicv_r2x = _bicv_trial(
345+
trial_seed = int(rng.integers(np.iinfo(np.int32).max))
346+
trial = _bicv_trial(
308347
X,
309348
rank,
310349
held_out_cell_frac,
311350
held_out_gene_frac,
312-
rng,
351+
trial_seed,
313352
tolerance,
314353
max_iter,
315354
compress,
316355
compression_kwarg,
317356
parafac2_kwarg,
318357
)
319358
rows.append(
320-
{"Rank": rank, "Repeat": repeat, "Metric": "BiCV R2X", "R2X": bicv_r2x}
359+
{
360+
"Rank": rank,
361+
"Repeat": repeat,
362+
"Metric": "BiCV R2X",
363+
"R2X": trial.pop("BiCV R2X"),
364+
**trial,
365+
}
321366
)
322367

323368
results = pd.DataFrame(rows)

‎scrise/tests/test_rank_selection.py‎

Lines changed: 97 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
import pandas as pd
88
import pytest
99

10-
from ..rank_selection import _train_cell_loadings, bicv
10+
from ..rank_selection import _bicv_trial, _train_cell_loadings, bicv
1111

1212

1313
def _make_test_data(
@@ -45,7 +45,7 @@ def test_bicv_shape_and_range():
4545
results = bicv(X, ranks, n_repeats=n_repeats, random_state=0, max_iter=50)
4646

4747
assert isinstance(results, pd.DataFrame)
48-
assert set(results.columns) == {"Rank", "Repeat", "Metric", "R2X"}
48+
assert {"Rank", "Repeat", "Metric", "R2X"} <= set(results.columns)
4949
assert set(results["Metric"]) == {"Fit R2X", "BiCV R2X"}
5050
assert set(results["Rank"]) == set(ranks)
5151

@@ -268,3 +268,98 @@ def test_bicv_runs_on_interleaved_conditions():
268268

269269
result = bicv(shuffled, ranks=[2, 3], n_repeats=1, random_state=0, max_iter=50)
270270
assert np.isfinite(result["R2X"]).all()
271+
272+
273+
# ---------------------------------------------------------------------------
274+
# Per-trial diagnostics (issue #548)
275+
# ---------------------------------------------------------------------------
276+
277+
_TRIAL_COLUMNS = [
278+
"Train Block R2X",
279+
"NTrainGenes",
280+
"NTestGenes",
281+
"NTrainCells",
282+
"NTestCells",
283+
"Seed",
284+
]
285+
286+
287+
def test_bicv_reports_per_trial_diagnostics():
288+
X = _make_test_data()
289+
results = bicv(X, [2, 3], n_repeats=2, random_state=0, max_iter=50)
290+
291+
assert set(_TRIAL_COLUMNS) <= set(results.columns)
292+
bicv_rows = results[results["Metric"] == "BiCV R2X"]
293+
assert bicv_rows[_TRIAL_COLUMNS].notna().all().all()
294+
295+
296+
def test_trial_columns_are_absent_on_the_unsplit_fit_rows():
297+
""" "Fit R2X" is a different fit on the full data, so it has no split."""
298+
X = _make_test_data()
299+
results = bicv(X, [2], n_repeats=1, random_state=0, max_iter=50)
300+
fit_rows = results[results["Metric"] == "Fit R2X"]
301+
assert fit_rows[_TRIAL_COLUMNS].isna().all().all()
302+
303+
304+
def test_reported_block_sizes_match_the_requested_split():
305+
X = _make_test_data()
306+
results = bicv(
307+
X,
308+
[2],
309+
n_repeats=2,
310+
random_state=0,
311+
max_iter=50,
312+
held_out_gene_frac=0.25,
313+
held_out_cell_frac=0.25,
314+
)
315+
rows = results[results["Metric"] == "BiCV R2X"]
316+
317+
assert (rows["NTrainGenes"] + rows["NTestGenes"] == X.n_vars).all()
318+
assert (rows["NTrainCells"] + rows["NTestCells"] == X.n_obs).all()
319+
# Genes split globally, so the count is exact rather than approximate.
320+
assert (rows["NTestGenes"] == round(X.n_vars * 0.25)).all()
321+
# Cells split within each condition, so allow for per-condition rounding.
322+
assert np.allclose(rows["NTestCells"] / X.n_obs, 0.25, atol=0.05)
323+
324+
325+
def test_train_block_r2x_is_in_sample_and_so_climbs_with_rank():
326+
"""The reason this column exists.
327+
328+
A BiCV curve is only interpretable against the in-sample fit on the *same*
329+
block: the useful signal is the held-out score turning over while this one
330+
keeps climbing. The separate "Fit R2X" metric is a different fit on
331+
different data and cannot play that role.
332+
"""
333+
X = _make_test_data()
334+
results = bicv(X, [1, 3, 6], n_repeats=2, random_state=0, max_iter=100)
335+
rows = results[results["Metric"] == "BiCV R2X"]
336+
337+
by_rank = rows.groupby("Rank")["Train Block R2X"].mean().sort_index()
338+
assert np.all(np.diff(by_rank.to_numpy()) >= -1e-6)
339+
assert (rows["Train Block R2X"] <= 1.0 + 1e-6).all()
340+
341+
342+
def test_reported_seed_reproduces_its_own_trial():
343+
"""A trial can be rerun in isolation from the seed the table reports."""
344+
X = _make_test_data()
345+
results = bicv(X, [3], n_repeats=3, random_state=0, max_iter=60)
346+
rows = results[results["Metric"] == "BiCV R2X"].reset_index(drop=True)
347+
348+
# Seeds differ across repeats, or "reproducible" would be vacuous.
349+
assert rows["Seed"].nunique() == len(rows)
350+
351+
target = rows.iloc[1]
352+
replayed = _bicv_trial(
353+
X,
354+
rank=3,
355+
held_out_cell_frac=0.2,
356+
held_out_gene_frac=0.2,
357+
seed=int(target["Seed"]),
358+
tolerance=1e-6,
359+
max_iter=60,
360+
)
361+
assert replayed["BiCV R2X"] == pytest.approx(target["R2X"], rel=1e-9)
362+
assert replayed["Train Block R2X"] == pytest.approx(
363+
target["Train Block R2X"], rel=1e-9
364+
)
365+
assert replayed["NTestGenes"] == target["NTestGenes"]

0 commit comments

Comments
 (0)