Skip to content

Commit 1b0b97f

Browse files
aarmeyAaron MeyerclaudeCopilot
authored
Add mkdocs documentation (#3)
* Add mkdocs documentation Adds an mkdocs-material site with an API reference (mkdocstrings) for every public module, matching the docs setup used across other meyer-lab Python repos (RISE, DDMC, parafac2). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCdnqsvzruAac1NCN9Thic * Add GitHub Pages deploy workflow for docs Builds the mkdocs site on push to main and uploads it as a Pages artifact (actions/upload-pages-artifact + deploy-pages), not a gh-pages branch, matching the pattern used by RISE/DDMC/parafac2. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCdnqsvzruAac1NCN9Thic * Restrict test workflow permissions Co-authored-by: aarmey <2065146+aarmey@users.noreply.github.com> --------- Co-authored-by: Aaron Meyer <git@ameyer.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent 728e8ac commit 1b0b97f

13 files changed

Lines changed: 794 additions & 1 deletion

File tree

‎.github/workflows/docs.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Docs
2+
3+
on:
4+
push:
5+
branches: [main]
6+
7+
permissions:
8+
contents: read
9+
pages: write
10+
id-token: write
11+
12+
concurrency:
13+
group: pages
14+
cancel-in-progress: false
15+
16+
jobs:
17+
build:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@v4
21+
- name: Install uv
22+
uses: astral-sh/setup-uv@v5
23+
with:
24+
enable-cache: true
25+
- name: Install dependencies
26+
run: uv sync --group docs
27+
- name: Build docs
28+
run: uv run mkdocs build --strict
29+
- name: Setup Pages
30+
uses: actions/configure-pages@v5
31+
- name: Upload artifact
32+
uses: actions/upload-pages-artifact@v3
33+
with:
34+
path: site
35+
36+
deploy:
37+
needs: build
38+
runs-on: ubuntu-latest
39+
environment:
40+
name: github-pages
41+
url: ${{ steps.deployment.outputs.page_url }}
42+
steps:
43+
- name: Deploy to GitHub Pages
44+
id: deployment
45+
uses: actions/deploy-pages@v4

‎.github/workflows/test.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ on:
44
push:
55
pull_request:
66

7+
permissions:
8+
contents: read
9+
710
jobs:
811
build:
912
runs-on: ubuntu-latest

‎docs/index.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# bootstraptools
2+
3+
> The population is to the sample as the sample is to the bootstrap samples
4+
5+
Local storage, query, and uncertainty-quantification infrastructure for
6+
**bootstrap experiments** that evaluate regression and classification model
7+
performance. A bootstrap runner writes one run (B replicates) to a local
8+
store, and downstream processing scripts query it back and compute point
9+
estimates and confidence intervals **on demand**.
10+
11+
`bootstraptools` is **model-agnostic**: it sees a model as a generic `fit` /
12+
`predict_proba` object and stores whatever scalars and named arrays a runner
13+
hands it. It is shaped around plain sklearn architectures, so other model
14+
types may need gentle massaging (a `fit(X, y)` / `predict_proba(X) -> (n,
15+
2)` shim and a way to slice examples) to plug in.
16+
17+
```bash
18+
uv sync # uv manages deps/venv
19+
uv run pytest -q # full test suite
20+
uv run python examples/reference_runner.py # end-to-end demo, prints CIs
21+
```
22+
23+
## Data model
24+
25+
| Concept | Meaning | On disk |
26+
|---|---|---|
27+
| **Store** | A root directory holding many runs. | `<store>/runs/` |
28+
| **Run** | One bootstrap experiment: B replicates for a `(dataset, model, procedure)` config, plus tags/config/summary. | `<store>/runs/<run_id>/` |
29+
| **Replicate** | One refit + evaluation: scalar metrics, per-sample predictions, resample membership, seeds, optional rich arrays / model state. | rows/files within a run |
30+
31+
Per-run layout:
32+
33+
```
34+
<store>/runs/<run_id>/
35+
meta.json # tags, config, procedure, dataset, model_label, seeds,
36+
# universe_size, status, n_replicates, summary
37+
replicates.parquet # 1 row / replicate: seeds, fit metadata, scalar metrics
38+
predictions.parquet # 1 row / (replicate, sample): sample_idx, y_true, p, split
39+
membership.parquet # 1 row / (replicate, sample, count>0)
40+
apparent.parquet # opt-in: the full-data model's predictions on all n rows
41+
arrays/replicate_XXXX.npz # opt-in rich per-replicate arrays
42+
model_state/replicate_XXXX.npz # opt-in full fitted params (store_model_state=True)
43+
```
44+
45+
Tables are [polars](https://github.com/pola-rs/polars)/parquet, arrays are
46+
npz. `arrays/` and `model_state/` are written **immediately** per replicate
47+
(durable if a long run crashes).
48+
49+
## Writing a run
50+
51+
`init` -> per-replicate `log_replicate` -> `finish`. The runner owns the
52+
fit/predict loop.
53+
54+
```python
55+
import bootstraptools as bs
56+
57+
seeds = bs.derive_seeds(RANDOM_SEED, ["dataset", "model", "bootstrap"])
58+
plans = bs.train_resample_holdout(
59+
train_idx, val_idx, universe_size=n, n_replicates=100,
60+
bootstrap_seed=seeds["bootstrap"],
61+
)
62+
fit_seeds = bs.replicate_seeds(seeds["model"], len(plans))
63+
64+
with bs.init(store, procedure="train_resample_holdout", dataset="bal",
65+
model_label="ULTRA",
66+
config={"rank": 4, "B": 100, "bootstrap_seed": seeds["bootstrap"]},
67+
tags=["fig3", "pooling"]) as run:
68+
for plan in plans:
69+
model.fit(bs.select(X, plan.fit_indices), y[plan.fit_indices])
70+
p = model.predict_proba(bs.select(X, plan.eval_indices))[:, 1]
71+
run.log_replicate(
72+
plan,
73+
metrics={"brier": ..., "auroc": ...},
74+
y_true=y[plan.eval_indices], p=p, # -> predictions.parquet
75+
arrays={"attn_entropy": ...}, # opt-in rich artifacts
76+
model_fit_seed=fit_seeds[plan.replicate_idx],
77+
)
78+
```
79+
80+
`log_replicate` pulls the replicate index / seed / membership from the
81+
`ResamplePlan`. There is no fixed metric schema; column names are whatever
82+
you pass. `bs.select(X, idx)` slices ndarrays (`X[idx]`) or lists of bags
83+
(`[X[i] for i in idx]`). See the [procedures reference](reference/procedures.md)
84+
for which procedure to pick, and the [resample reference](reference/resample.md)
85+
for the variable-size draw primitives beneath them.
86+
87+
## Querying results
88+
89+
```python
90+
runs = bs.query_runs(store, {"tags": "fig3", "dataset": "bal"}) # 1 row / run
91+
tbl = bs.query_run_table(store, run_id, table="replicates") # one run's table
92+
all_ = bs.load_runs_table(store, {"tags": "fig3"}, table="replicates") # concat + run_id col
93+
membership = bs.membership_matrix(store, run_id) # dense (B, n) counts (for BCa)
94+
```
95+
96+
`query_runs` filters (AND-ed) on `procedure`/`dataset`/`model_label`/`status`/
97+
`run_id`, on `tags`, and on any `config.<key>`. config/summary flatten into
98+
`config.*` / `summary.*` columns.
99+
100+
## Comparing across runs (paired analyses)
101+
102+
Represent any "same resample, different fit" axis as **separate runs
103+
sharing the `bootstrap_seed`**: identical seeds give identical per-replicate
104+
`fit_indices`, so the runs' replicates are row-joinable on `replicate_idx`
105+
for paired comparisons.
106+
107+
## Where to go next
108+
109+
Browse the [API reference](reference/resample.md) for details on every
110+
public function and class, or read `DOCS.md` in the repository for a deeper
111+
walkthrough of the bootstrap procedures and reproducibility model.

‎docs/reference/optimism.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.optimism`
2+
3+
::: bootstraptools.optimism

‎docs/reference/procedures.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.procedures`
2+
3+
::: bootstraptools.procedures

‎docs/reference/query.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.query`
2+
3+
::: bootstraptools.query

‎docs/reference/resample.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.resample`
2+
3+
::: bootstraptools.resample

‎docs/reference/seeds.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.seeds`
2+
3+
::: bootstraptools.seeds

‎docs/reference/store.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.store`
2+
3+
::: bootstraptools.store

‎docs/reference/uq.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# `bootstraptools.uq`
2+
3+
::: bootstraptools.uq

0 commit comments

Comments
 (0)