Skip to content

Commit 5db609d

Browse files
committed
feat(stack-encrypt): size a match query's fetch with target_limit
ADR-0009 decision 5: a caller who wants `want` real matches from a probabilistic query fetches `want + Poisson_quantile(λ = p·N, confidence)` rows, where `p` is the mean over stored filters of `f^b`, `f` a filter's fill and `b` the bits the query sets. - `Probabilistic::target_limit(&self, want, &Stats) -> Limit`. - `op::Stats::declared(rows)` uses a new `MatchConfig::expected_tokens()` prior, a default method returning `None`, so existing configurations compile unchanged; it is not part of the index's identity. - `op::Stats::measured(rows, histogram)` takes `(set bits, rows)` buckets from one sampled aggregate over the column, and takes the mean of the power, never the power of the mean fill. An empty histogram falls back to the prior. - `op::Limit` is `Rows(n)` or `Unbounded`, the latter when nothing is known of the stored filters or the confidence is not in (0, 1). `fetch_or(page)` reads either as a fetch size. - The quantile sums the Poisson mass exactly in log space from `λ − 10√λ`, so it costs about `12√λ` terms; past `λ = 1e7` it is the conservative `λ + 8√λ`. Unit tests reproduce the ADR's worked table and its mixed-length column, and pin the quantile to reference values computed with `lgamma`. Closes #1184 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
1 parent 2d33d6c commit 5db609d

8 files changed

Lines changed: 560 additions & 16 deletions

File tree

‎packages/stack-encrypt/CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3636
of the text. An exact term has no `verify`, and none of this crate's
3737
indexes can answer `op::Matches` but with a probabilistic term
3838
(`op::TermFor`).
39+
- **`target_limit` says how many rows a match query should fetch.**
40+
`query.target_limit(want, &stats)` is `want` plus the false positives
41+
expected at the confidence `stats` names (99% by default), as an
42+
`op::Limit`. `op::Stats::declared(rows)` uses the new
43+
`MatchConfig::expected_tokens()` prior; `op::Stats::measured(rows,
44+
histogram)` uses a histogram of the stored filters' set-bit counts. With
45+
neither, the limit is `Limit::Unbounded`, and the caller pages with a
46+
cursor.
3947

4048
- **Errors now expose stable codes, help text, and structured details.**
4149
All error types implement `miette::Diagnostic` with codes such as

‎packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md‎

Lines changed: 22 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -344,9 +344,10 @@ takes them as numbers in `Stats` and does not care where they came from.
344344
There are two sources, and a column has the first one from the day it is
345345
declared.
346346

347-
**Declared.** A match index declares `expected_tokens`, the typical token
348-
count of the values it will hold, beside `m` and `k`: 20 for a name column,
349-
200 for a notes column. From it `Stats::declared(rows)` takes
347+
**Declared.** A match configuration declares `expected_tokens()`, the typical
348+
token count of the values it will hold, beside the `m` and `k` of its
349+
options: 20 for a name column, 200 for a notes column. It is not part of the
350+
index's identity and changes no term. From it `Stats::declared(rows)` takes
350351
`f = 1 − e^(−k·expected_tokens/m)` and `p = f^b`. This is a prior, and
351352
`rows` is a planning estimate too. It is enough to warn before a query runs
352353
(a three-letter needle against a notes column) and to size a first fetch; it
@@ -369,8 +370,16 @@ query has. The sample bounds the cost on a large table; `TABLESAMPLE` is a
369370
page sample, which is fine for a distribution and unsuited to an exact count.
370371
It is also a sample of pages, so on a table of a few thousand rows a 1%
371372
sample can select none; below that size the clause is dropped and the whole
372-
table is read, which costs nothing worth saving. `rows` comes from `SELECT reltuples FROM pg_class WHERE oid =
373-
'users'::regclass`, a catalog read.
373+
table is read, which costs nothing worth saving. `rows` comes from
374+
`SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass`, a catalog
375+
read. An empty histogram falls back to the declared prior.
376+
377+
`target_limit` returns a `Limit`: `Rows(n)`, or `Unbounded` when the
378+
statistics say nothing about the stored filters (no prior and no
379+
histogram) or the confidence is not below 1. `fetch_or(page)` reads either
380+
as a fetch size, for a caller that pages anyway. Past ten million expected
381+
false positives the quantile is the conservative `λ + 8√λ`, since the query
382+
is hopeless at that point.
374383

375384
Where the numbers live and who keeps them in step was asked on the review,
376385
so it is written down. They are read from the server by the application, at
@@ -431,7 +440,7 @@ shot, with a bound, and the completeness check from decision 5:
431440

432441
```rust
433442
let term = cipher.query("ali").using(&name_plan).op(op::Matches::new()).await?;
434-
let fetch = term.target_limit(10, &stats).fetch();
443+
let fetch = term.target_limit(10, &stats).fetch_or(50);
435444

436445
let rows: Vec<UserRow> = sqlx::query_as(
437446
"SELECT id, name FROM users WHERE name @@ $1 ORDER BY id LIMIT $2",
@@ -669,7 +678,7 @@ the operator is `@@`, column on the left. The query selects the ciphertext
669678

670679
```rust
671680
let term = cipher.query("ali").using(&name_plan).op(op::Matches::new()).await?;
672-
let fetch = term.target_limit(10, &stats).fetch();
681+
let fetch = term.target_limit(10, &stats).fetch_or(50);
673682

674683
let rows: Vec<UserRow> = sqlx::query_as(
675684
"SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2",
@@ -691,12 +700,12 @@ The `Stats` it used were read once, at startup, by the application:
691700
```rust
692701
let rows: i64 = sqlx::query_scalar("SELECT reltuples::bigint FROM pg_class WHERE oid = 'users'::regclass")
693702
.fetch_one(&pool).await?;
694-
let histogram: Vec<(i16, i64)> = sqlx::query_as(
703+
let histogram: Vec<(i32, i64)> = sqlx::query_as(
695704
"SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*)
696705
FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1",
697706
)
698707
.fetch_all(&pool).await?;
699-
let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u16, n as u64)));
708+
let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u32, n as u64)));
700709
```
701710

702711
or, before anyone has measured, `Stats::declared(1_000_000)`, which uses the
@@ -716,7 +725,7 @@ let rows: Vec<UserRow> = sqlx::query_as(
716725
)
717726
.bind(sqlx::types::Json(&name))
718727
.bind(sqlx::types::Json(&age))
719-
.bind(name.target_limit(10, &stats).fetch() as i64)
728+
.bind(name.target_limit(10, &stats).fetch_or(50) as i64)
720729
.fetch_all(&pool).await?;
721730
let hits: Vec<User> = cipher.decrypt_all(rows).await?
722731
.into_iter().filter(|u| name.verify(&u.name)).take(10).collect();
@@ -858,7 +867,7 @@ it binds as a parameter in any library built on `database/sql` or pgx:
858867

859868
```go
860869
q, _ := users.Fields.Name.Matches(ctx, cipher, "ali")
861-
limit := q.TargetLimit(10, stats).Fetch()
870+
limit := q.TargetLimit(10, stats).FetchOr(50)
862871

863872
rows, err := db.QueryContext(ctx,
864873
`SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2`,
@@ -964,7 +973,7 @@ TypeScript programs use:
964973

965974
```ts
966975
// pg
967-
const limit = name.targetLimit(10, stats).fetch()
976+
const limit = name.targetLimit(10, stats).fetchOr(50)
968977
const { rows } = await pool.query(
969978
'SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2',
970979
[name, limit],
@@ -1010,7 +1019,7 @@ const q = await ops.matches(users.name, 'ali') // { sql, verify, target
10101019
const rows = await db.select().from(users)
10111020
.where(q.sql) // name @@ '…'::eql_v4.query_text_match
10121021
.orderBy(users.id)
1013-
.limit(q.targetLimit(10, stats).fetch())
1022+
.limit(q.targetLimit(10, stats).fetchOr(50))
10141023
const hits = (await cipher.decrypt(users, rows)).filter((u) => q.verify(u.name)).slice(0, 10)
10151024
```
10161025

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ use std::marker::PhantomData;
5555

5656
use stack_kms::MaybeSend;
5757

58+
pub(crate) mod stats;
59+
pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE};
60+
5861
use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, Needle};
5962
use crate::target::{Equality, Index, Match};
6063
use crate::PlanError;
@@ -202,6 +205,19 @@ pub trait Probabilistic: QueryTerm {
202205
/// the index was declared over, so every outcome other than a match is
203206
/// `false`, never an error.
204207
fn verify(&self, value: &Self::Plaintext) -> bool;
208+
209+
/// How many candidate rows to fetch so that, at the confidence `stats`
210+
/// names, `want` of them [`verify`](Self::verify), when the table holds
211+
/// that many matches.
212+
///
213+
/// A probabilistic bound, not a guarantee: at 99% confidence one query
214+
/// in a hundred draws more false positives than it allows for. The
215+
/// candidate count says which case a fetch is in. If the server
216+
/// returned fewer rows than the limit, the predicate is exhausted, and
217+
/// fewer than `want` verified rows means fewer than `want` matches. If
218+
/// it returned exactly the limit, the result is a page: carry on from
219+
/// its last row with a cursor.
220+
fn target_limit(&self, want: u64, stats: &Stats) -> Limit;
205221
}
206222

207223
/// The query terms that may answer the operator `Op`: what

0 commit comments

Comments
 (0)