|
| 1 | +# bootstraptools |
| 2 | + |
| 3 | +> The population is to the sample as the sample is to the bootstrap samples |
| 4 | +
|
| 5 | +Local storage, query, and uncertainty-quantification infrastructure for |
| 6 | +**bootstrap experiments** that evaluate regression and classification model |
| 7 | +performance. A bootstrap runner writes one run (B replicates) to a local |
| 8 | +store, and downstream processing scripts query it back and compute point |
| 9 | +estimates and confidence intervals **on demand**. |
| 10 | + |
| 11 | +`bootstraptools` is **model-agnostic**: it sees a model as a generic `fit` / |
| 12 | +`predict_proba` object and stores whatever scalars and named arrays a runner |
| 13 | +hands it. It is shaped around plain sklearn architectures, so other model |
| 14 | +types may need gentle massaging (a `fit(X, y)` / `predict_proba(X) -> (n, |
| 15 | +2)` shim and a way to slice examples) to plug in. |
| 16 | + |
| 17 | +```bash |
| 18 | +uv sync # uv manages deps/venv |
| 19 | +uv run pytest -q # full test suite |
| 20 | +uv run python examples/reference_runner.py # end-to-end demo, prints CIs |
| 21 | +``` |
| 22 | + |
| 23 | +## Data model |
| 24 | + |
| 25 | +| Concept | Meaning | On disk | |
| 26 | +|---|---|---| |
| 27 | +| **Store** | A root directory holding many runs. | `<store>/runs/` | |
| 28 | +| **Run** | One bootstrap experiment: B replicates for a `(dataset, model, procedure)` config, plus tags/config/summary. | `<store>/runs/<run_id>/` | |
| 29 | +| **Replicate** | One refit + evaluation: scalar metrics, per-sample predictions, resample membership, seeds, optional rich arrays / model state. | rows/files within a run | |
| 30 | + |
| 31 | +Per-run layout: |
| 32 | + |
| 33 | +``` |
| 34 | +<store>/runs/<run_id>/ |
| 35 | + meta.json # tags, config, procedure, dataset, model_label, seeds, |
| 36 | + # universe_size, status, n_replicates, summary |
| 37 | + replicates.parquet # 1 row / replicate: seeds, fit metadata, scalar metrics |
| 38 | + predictions.parquet # 1 row / (replicate, sample): sample_idx, y_true, p, split |
| 39 | + membership.parquet # 1 row / (replicate, sample, count>0) |
| 40 | + apparent.parquet # opt-in: the full-data model's predictions on all n rows |
| 41 | + arrays/replicate_XXXX.npz # opt-in rich per-replicate arrays |
| 42 | + model_state/replicate_XXXX.npz # opt-in full fitted params (store_model_state=True) |
| 43 | +``` |
| 44 | + |
| 45 | +Tables are [polars](https://github.com/pola-rs/polars)/parquet, arrays are |
| 46 | +npz. `arrays/` and `model_state/` are written **immediately** per replicate |
| 47 | +(durable if a long run crashes). |
| 48 | + |
| 49 | +## Writing a run |
| 50 | + |
| 51 | +`init` -> per-replicate `log_replicate` -> `finish`. The runner owns the |
| 52 | +fit/predict loop. |
| 53 | + |
| 54 | +```python |
| 55 | +import bootstraptools as bs |
| 56 | + |
| 57 | +seeds = bs.derive_seeds(RANDOM_SEED, ["dataset", "model", "bootstrap"]) |
| 58 | +plans = bs.train_resample_holdout( |
| 59 | + train_idx, val_idx, universe_size=n, n_replicates=100, |
| 60 | + bootstrap_seed=seeds["bootstrap"], |
| 61 | +) |
| 62 | +fit_seeds = bs.replicate_seeds(seeds["model"], len(plans)) |
| 63 | + |
| 64 | +with bs.init(store, procedure="train_resample_holdout", dataset="bal", |
| 65 | + model_label="ULTRA", |
| 66 | + config={"rank": 4, "B": 100, "bootstrap_seed": seeds["bootstrap"]}, |
| 67 | + tags=["fig3", "pooling"]) as run: |
| 68 | + for plan in plans: |
| 69 | + model.fit(bs.select(X, plan.fit_indices), y[plan.fit_indices]) |
| 70 | + p = model.predict_proba(bs.select(X, plan.eval_indices))[:, 1] |
| 71 | + run.log_replicate( |
| 72 | + plan, |
| 73 | + metrics={"brier": ..., "auroc": ...}, |
| 74 | + y_true=y[plan.eval_indices], p=p, # -> predictions.parquet |
| 75 | + arrays={"attn_entropy": ...}, # opt-in rich artifacts |
| 76 | + model_fit_seed=fit_seeds[plan.replicate_idx], |
| 77 | + ) |
| 78 | +``` |
| 79 | + |
| 80 | +`log_replicate` pulls the replicate index / seed / membership from the |
| 81 | +`ResamplePlan`. There is no fixed metric schema; column names are whatever |
| 82 | +you pass. `bs.select(X, idx)` slices ndarrays (`X[idx]`) or lists of bags |
| 83 | +(`[X[i] for i in idx]`). See the [procedures reference](reference/procedures.md) |
| 84 | +for which procedure to pick, and the [resample reference](reference/resample.md) |
| 85 | +for the variable-size draw primitives beneath them. |
| 86 | + |
| 87 | +## Querying results |
| 88 | + |
| 89 | +```python |
| 90 | +runs = bs.query_runs(store, {"tags": "fig3", "dataset": "bal"}) # 1 row / run |
| 91 | +tbl = bs.query_run_table(store, run_id, table="replicates") # one run's table |
| 92 | +all_ = bs.load_runs_table(store, {"tags": "fig3"}, table="replicates") # concat + run_id col |
| 93 | +membership = bs.membership_matrix(store, run_id) # dense (B, n) counts (for BCa) |
| 94 | +``` |
| 95 | + |
| 96 | +`query_runs` filters (AND-ed) on `procedure`/`dataset`/`model_label`/`status`/ |
| 97 | +`run_id`, on `tags`, and on any `config.<key>`. config/summary flatten into |
| 98 | +`config.*` / `summary.*` columns. |
| 99 | + |
| 100 | +## Comparing across runs (paired analyses) |
| 101 | + |
| 102 | +Represent any "same resample, different fit" axis as **separate runs |
| 103 | +sharing the `bootstrap_seed`**: identical seeds give identical per-replicate |
| 104 | +`fit_indices`, so the runs' replicates are row-joinable on `replicate_idx` |
| 105 | +for paired comparisons. |
| 106 | + |
| 107 | +## Where to go next |
| 108 | + |
| 109 | +Browse the [API reference](reference/resample.md) for details on every |
| 110 | +public function and class, or read `DOCS.md` in the repository for a deeper |
| 111 | +walkthrough of the bootstrap procedures and reproducibility model. |
0 commit comments