Skip to content

Commit b3dd1f7

Browse files
committed
fix(stack-encrypt): address the review of target_limit
- The Poisson quantile is a bounded, normalised sum over `[λ − 10√λ, λ + 12√λ + 50]`. Each term is a log relative to the window's first, so a large `λ` no longer loses the starting term to cancellation, and the sums run from the top, where the terms are smallest, so a confidence near 1 is compared against a tail rounding did not swallow. It matches an exact reference to within one up to 1 − 1e-12 (was off by thousands at λ ≈ 4.75e6), and no mutation of it can loop forever. `ln_factorial` is gone. - A `Limit` never exceeds `op::MAX_ROWS` (`i64::MAX`), the largest SQL `LIMIT`; `reltuples` of −1 read as a `u64` no longer overflows a bind. The ADR's statements use `greatest(reltuples, 0)`. - Histogram counts are summed as floats, so huge counts cannot overflow. - The independence assumption is stated in ADR-0009 decision 5 and on `target_limit`: a row sharing tokens with the needle passes more often, so the confidence is nominal on natural text; `verify` still decides. - Docs: the stats SQL filters NULL values; `target_limit` says a fetch that returned the limit is a page only when fewer than `want` verified; `expected_tokens` counts distinct tokens; decision 4's code shows `target_limit` on `Probabilistic`. - Tests: exact references at high confidence and at λ = 1e7 (which kills the threshold mutant), the `MAX_ROWS` cap, float-summed counts, and the measured-column test bounds the fetch against the histogram's own λ. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
1 parent 5db609d commit b3dd1f7

6 files changed

Lines changed: 138 additions & 68 deletions

File tree

‎packages/stack-encrypt/CHANGELOG.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4343
`MatchConfig::expected_tokens()` prior; `op::Stats::measured(rows,
4444
histogram)` uses a histogram of the stored filters' set-bit counts. With
4545
neither, the limit is `Limit::Unbounded`, and the caller pages with a
46-
cursor.
46+
cursor. A limit never exceeds `op::MAX_ROWS` (`i64::MAX`), the largest
47+
SQL `LIMIT`. The bound assumes a non-matching row shares no token with
48+
the needle, so on natural text its confidence is nominal; `verify` still
49+
decides every row.
4750

4851
- **Errors now expose stable codes, help text, and structured details.**
4952
All error types implement `miette::Diagnostic` with codes such as

‎packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md‎

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -223,8 +223,7 @@ pub trait QueryTerm {}
223223
pub trait Probabilistic: QueryTerm {
224224
type Plaintext: ?Sized;
225225
fn verify(&self, value: &Self::Plaintext) -> bool;
226-
// `target_limit(&self, want, &Stats) -> Limit` joins it with decision 5
227-
// (#1184).
226+
fn target_limit(&self, want: u64, stats: &Stats) -> Limit;
228227
}
229228

230229
pub trait Ranked: QueryTerm {
@@ -318,6 +317,15 @@ positives); the mean fill, 0.25, raised to 6 predicts 245. The number the
318317
bound needs is therefore `p = mean(f_i^b)`, and the statistics the client
319318
holds must let it compute that for the `b` of each query.
320319

320+
The model assumes a non-matching row shares no token with the needle. The
321+
same token sets the same bits in every filter, so a row holding part of the
322+
needle has fewer bits left to chance: "police" already holds two of
323+
"alice"'s three 3-grams, and passes at `f^6` rather than `f^18`. On natural
324+
text the stated confidence is therefore nominal, and a measured histogram
325+
cannot correct it, because it measures fill, not overlap. Correctness does
326+
not depend on it: `verify` decides every row, and the candidate count says
327+
whether a fetch was complete.
328+
321329
False positives are then `Poisson(λ = p·N)`, and the worst case that still
322330
satisfies the caller is exactly `want` true matches preceded by every false
323331
positive, so
@@ -361,6 +369,7 @@ fills. One aggregate gives it, at most `m` buckets:
361369
```sql
362370
SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*)
363371
FROM users TABLESAMPLE SYSTEM (1)
372+
WHERE name IS NOT NULL
364373
GROUP BY 1
365374
```
366375

@@ -371,8 +380,10 @@ page sample, which is fine for a distribution and unsuited to an exact count.
371380
It is also a sample of pages, so on a table of a few thousand rows a 1%
372381
sample can select none; below that size the clause is dropped and the whole
373382
table is read, which costs nothing worth saving. `rows` comes from
374-
`SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass`, a catalog
375-
read. An empty histogram falls back to the declared prior.
383+
`SELECT greatest(reltuples, 0) FROM pg_class WHERE oid = 'users'::regclass`,
384+
a catalog read; `reltuples` is −1 for a table that has never been vacuumed
385+
or analysed, hence the `greatest`. A `Limit` never exceeds `i64::MAX`, the
386+
largest `LIMIT` PostgreSQL takes. An empty histogram falls back to the declared prior.
376387

377388
`target_limit` returns a `Limit`: `Rows(n)`, or `Unbounded` when the
378389
statistics say nothing about the stored filters (no prior and no
@@ -698,11 +709,11 @@ let hits: Vec<User> = cipher.decrypt_all(rows).await?
698709
The `Stats` it used were read once, at startup, by the application:
699710

700711
```rust
701-
let rows: i64 = sqlx::query_scalar("SELECT reltuples::bigint FROM pg_class WHERE oid = 'users'::regclass")
712+
let rows: i64 = sqlx::query_scalar("SELECT greatest(reltuples, 0)::bigint FROM pg_class WHERE oid = 'users'::regclass")
702713
.fetch_one(&pool).await?;
703714
let histogram: Vec<(i32, i64)> = sqlx::query_as(
704715
"SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*)
705-
FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1",
716+
FROM users TABLESAMPLE SYSTEM (1) WHERE name IS NOT NULL GROUP BY 1",
706717
)
707718
.fetch_all(&pool).await?;
708719
let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u32, n as u64)));
@@ -1033,8 +1044,8 @@ application, or from the declared prior:
10331044

10341045
```ts
10351046
const stats = Stats.measured(
1036-
(await pool.query("SELECT reltuples::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows,
1037-
(await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1')).rows,
1047+
(await pool.query("SELECT greatest(reltuples, 0)::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows,
1048+
(await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) WHERE name IS NOT NULL GROUP BY 1')).rows,
10381049
)
10391050
const prior = Stats.declared(1_000_000) // uses users.name's expectedTokens
10401051
```

‎packages/stack-encrypt/src/op/mod.rs‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ use std::marker::PhantomData;
5656
use stack_kms::MaybeSend;
5757

5858
pub(crate) mod stats;
59-
pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE};
59+
pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE, MAX_ROWS};
6060

6161
use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, Needle};
6262
use crate::target::{Equality, Index, Match};
@@ -211,12 +211,14 @@ pub trait Probabilistic: QueryTerm {
211211
/// that many matches.
212212
///
213213
/// A probabilistic bound, not a guarantee: at 99% confidence one query
214-
/// in a hundred draws more false positives than it allows for. The
215-
/// candidate count says which case a fetch is in. If the server
214+
/// in a hundred draws more false positives than it allows for, and the
215+
/// model assumes a non-matching row shares no token with the needle,
216+
/// which natural text does not honour, so the confidence is nominal.
217+
/// The candidate count says which case a fetch is in. If the server
216218
/// returned fewer rows than the limit, the predicate is exhausted, and
217219
/// fewer than `want` verified rows means fewer than `want` matches. If
218-
/// it returned exactly the limit, the result is a page: carry on from
219-
/// its last row with a cursor.
220+
/// it returned exactly the limit and fewer than `want` verified, the
221+
/// result is a page: carry on from its last row with a cursor.
220222
fn target_limit(&self, want: u64, stats: &Stats) -> Limit;
221223
}
222224

‎packages/stack-encrypt/src/op/stats.rs‎

Lines changed: 92 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -46,11 +46,16 @@ enum Fill {
4646
/// The confidence a [`Stats`] targets unless told otherwise.
4747
pub const DEFAULT_CONFIDENCE: f64 = 0.99;
4848

49+
/// The largest fetch a [`Limit`] names: PostgreSQL's `LIMIT` is a signed
50+
/// 64-bit integer.
51+
pub const MAX_ROWS: u64 = i64::MAX as u64;
52+
4953
impl Stats {
5054
/// Statistics from the index's declared prior
5155
/// ([`MatchConfig::expected_tokens`](crate::sem::MatchConfig::expected_tokens))
5256
/// and an estimate of the rows the query scans, such as
53-
/// `pg_class.reltuples`.
57+
/// `greatest(pg_class.reltuples, 0)` (`reltuples` is −1 for a table
58+
/// never analysed).
5459
///
5560
/// The prior knows nothing about the spread of value lengths, so it
5661
/// underestimates a column of mixed lengths; measure the column
@@ -74,6 +79,7 @@ impl Stats {
7479
/// ```sql
7580
/// SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*)
7681
/// FROM users TABLESAMPLE SYSTEM (1)
82+
/// WHERE name IS NOT NULL
7783
/// GROUP BY 1
7884
/// ```
7985
///
@@ -130,16 +136,18 @@ impl Stats {
130136
return Limit::Unbounded;
131137
}
132138
let lambda = p.clamp(0.0, 1.0) * self.rows as f64;
133-
Limit::Rows(want.saturating_add(poisson_quantile(lambda, self.confidence)))
139+
let rows = want.saturating_add(poisson_quantile(lambda, self.confidence));
140+
Limit::Rows(rows.min(MAX_ROWS))
134141
}
135142
}
136143

137144
/// The mean of `(bits_i / m)^b` over a histogram of set-bit counts, or
138145
/// `None` for an empty one. A count above `m` (a histogram from another
139146
/// configuration) is a full filter.
140147
fn mean_power(histogram: &[(u32, u64)], b: u32, m: u32) -> Option<f64> {
141-
let rows: u64 = histogram.iter().map(|&(_, rows)| rows).sum();
142-
if rows == 0 || m == 0 {
148+
// Summed as floats: counts from a caller need not fit a `u64` total.
149+
let rows: f64 = histogram.iter().map(|&(_, rows)| rows as f64).sum();
150+
if rows == 0.0 || m == 0 {
143151
return None;
144152
}
145153
let total: f64 = histogram
@@ -149,7 +157,7 @@ fn mean_power(histogram: &[(u32, u64)], b: u32, m: u32) -> Option<f64> {
149157
rows as f64 * fill.powi(b as i32)
150158
})
151159
.sum();
152-
Some(total / rows as f64)
160+
Some(total / rows)
153161
}
154162

155163
/// The fill of one stored filter of `tokens` tokens, each setting `k` of
@@ -158,7 +166,8 @@ pub(crate) fn expected_fill(tokens: u32, k: usize, m: u32) -> f64 {
158166
1.0 - (-(k as f64) * f64::from(tokens) / f64::from(m)).exp()
159167
}
160168

161-
/// How many rows a probabilistic query should fetch.
169+
/// How many rows a probabilistic query should fetch. Never more than
170+
/// [`MAX_ROWS`], the largest `LIMIT` PostgreSQL takes.
162171
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
163172
pub enum Limit {
164173
/// Fetch this many rows: `want` and the false positives expected at the
@@ -188,47 +197,58 @@ impl Limit {
188197

189198
/// The smallest `q` with `P(X ≤ q) ≥ confidence` for `X ~ Poisson(λ)`.
190199
///
191-
/// Exact up to floating point: the probabilities are summed in log space
192-
/// from where the mass begins, `λ − 10√λ` for a large `λ` (the mass below
193-
/// it is under `1e-23`), so the work is about `12√λ` terms. Past `1e7`
194-
/// expected false positives the query is hopeless anyway and the answer is
195-
/// the conservative `λ + 8√λ`, which covers any confidence below
200+
/// The probabilities are summed over the window `[λ − 10√λ, λ + 12√λ + 50]`,
201+
/// which holds all but about `1e-23` of the mass, and normalised by the
202+
/// window's total. Each term is kept as a log relative to the window's
203+
/// first, so no term carries the cancellation of `−λ + k·ln λ − ln k!` at a
204+
/// large `λ`, and the sums run from the small end, so a confidence close to
205+
/// 1 compares against a tail summed without losing it to rounding. The
206+
/// answer is exact to within one at any confidence below `1 − 1e-12`. The
207+
/// work is about `22√λ` terms.
208+
/// Past `1e7` expected false positives the query is hopeless anyway and the
209+
/// answer is the conservative `λ + 8√λ`, which covers any confidence below
196210
/// `1 − 1e-15`.
211+
///
212+
/// `λ ≤ 0` and NaN have a window of one term at 0, so the answer is 0.
197213
pub(crate) fn poisson_quantile(lambda: f64, confidence: f64) -> u64 {
198-
if lambda <= 0.0 || lambda.is_nan() {
199-
return 0;
200-
}
201214
if lambda > 1e7 {
202215
return (lambda + 8.0 * lambda.sqrt()).ceil() as u64;
203216
}
204-
// Saturates to 0 for a small `λ`, where the mass starts at 0.
205-
let start = (lambda - 10.0 * lambda.sqrt()).floor().max(0.0) as u64;
217+
let spread = lambda.sqrt();
218+
// A float-to-integer cast saturates, and maps NaN to 0.
219+
let start = (lambda - 10.0 * spread).floor().max(0.0) as u64;
220+
let end = (lambda + 12.0 * spread + 50.0).ceil() as u64;
206221
let ln_lambda = lambda.ln();
207-
let mut log_term = -lambda + start as f64 * ln_lambda - ln_factorial(start);
208-
let mut cdf = 0.0;
209-
let mut k = start;
210-
loop {
211-
cdf += log_term.exp();
212-
// Past the mode, a term this small adds nothing a float can hold:
213-
// the remaining mass is below any confidence that is not 1.
214-
if cdf >= confidence || (k as f64 > lambda && log_term < -50.0) {
215-
return k;
222+
let mut weights = Vec::with_capacity((end - start + 1) as usize);
223+
let mut log_term = 0.0;
224+
for k in start..=end {
225+
if k > start {
226+
log_term += ln_lambda - (k as f64).ln();
216227
}
217-
k += 1;
218-
log_term += ln_lambda - (k as f64).ln();
228+
weights.push(log_term);
219229
}
220-
}
221-
222-
/// `ln(n!)`: summed exactly below 20, Stirling's series above, where it is
223-
/// accurate to about `1e-12`.
224-
fn ln_factorial(n: u64) -> f64 {
225-
if n < 20 {
226-
return (2..=n).map(|i| (i as f64).ln()).sum();
230+
let peak = weights.iter().copied().fold(f64::NEG_INFINITY, f64::max);
231+
for weight in &mut weights {
232+
*weight = (*weight - peak).exp();
233+
}
234+
// Summed from the top down, where the terms are smallest, so the tail a
235+
// high confidence compares against is not lost to rounding.
236+
let total: f64 = weights.iter().rev().sum();
237+
let allowed = (1.0 - confidence) * total;
238+
// Walking down from the top, `above` is the mass strictly above `k`.
239+
// The answer is the lowest `k` whose mass above is within what the
240+
// confidence allows.
241+
let mut answer = end;
242+
let mut above = 0.0;
243+
for (offset, weight) in weights.iter().enumerate().rev() {
244+
if above <= allowed {
245+
answer = start + offset as u64;
246+
above += weight;
247+
} else {
248+
break;
249+
}
227250
}
228-
let x = n as f64 + 1.0;
229-
(x - 0.5) * x.ln() - x + 0.5 * (2.0 * std::f64::consts::PI).ln() + 1.0 / (12.0 * x)
230-
- 1.0 / (360.0 * x.powi(3))
231-
+ 1.0 / (1260.0 * x.powi(5))
251+
answer
232252
}
233253

234254
#[cfg(test)]
@@ -272,18 +292,29 @@ mod tests {
272292

273293
#[test]
274294
fn a_confidence_at_one_still_ends() {
275-
// A float CDF may never reach 1; the tail cut-off ends the sum.
295+
// A float CDF may never reach 1; the window's end ends the sum.
276296
let q = poisson_quantile(5.0, 1.0);
277-
assert!(q > 20, "{q}");
297+
assert!((20..=5 + 27 + 50).contains(&q), "{q}");
298+
assert_eq!(poisson_quantile(f64::NAN, 0.99), 0);
278299
}
279300

301+
/// High confidences over a large `λ`, where summing from `e^(−λ)`
302+
/// loses precision: the window is normalised instead. References are
303+
/// exact (mpmath's regularised incomplete gamma); the sum may land one
304+
/// above, never below.
280305
#[test]
281-
fn ln_factorial_is_continuous_where_the_series_takes_over() {
282-
let exact: f64 = (2..=20u64).map(|i| (i as f64).ln()).sum();
283-
assert!((ln_factorial(20) - exact).abs() < 1e-10);
284-
assert_eq!(ln_factorial(0), 0.0);
285-
assert_eq!(ln_factorial(1), 0.0);
286-
assert!((ln_factorial(5) - 120f64.ln()).abs() < 1e-12);
306+
fn the_quantile_holds_at_high_confidence_and_large_lambda() {
307+
for (lambda, confidence, exact) in [
308+
(3.4e6, 0.9999, 3_406_860),
309+
(4_750_922.0, 1.0 - 1e-12, 4_766_263),
310+
(1e7, 0.99, 10_007_357),
311+
] {
312+
let q = poisson_quantile(lambda, confidence);
313+
assert!(
314+
q == exact || q == exact + 1,
315+
"λ = {lambda}, confidence = {confidence}: {q}, exact {exact}"
316+
);
317+
}
287318
}
288319

289320
/// ADR-0009 decision 5's worked table: EQL v4 defaults (`k = 6`,
@@ -372,10 +403,23 @@ mod tests {
372403
}
373404

374405
#[test]
375-
fn want_does_not_overflow() {
406+
fn a_limit_fits_a_sql_limit() {
376407
assert_eq!(
377408
Stats::declared(10).limit(u64::MAX, Some(0.5)),
378-
Limit::Rows(u64::MAX)
409+
Limit::Rows(MAX_ROWS)
410+
);
411+
// `reltuples` is −1 for a table never analysed; read as a `u64` it is
412+
// the largest count there is.
413+
assert_eq!(
414+
Stats::declared(u64::MAX).limit(10, Some(0.5)),
415+
Limit::Rows(MAX_ROWS)
379416
);
380417
}
418+
419+
#[test]
420+
fn histogram_counts_do_not_overflow() {
421+
let stats = Stats::measured(10, [(1024, u64::MAX), (2048, u64::MAX)]);
422+
let p = stats.pass_rate(1, 2048, None).unwrap();
423+
assert!((p - 0.75).abs() < 1e-12, "{p}");
424+
}
381425
}

‎packages/stack-encrypt/src/sem/mod.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -472,8 +472,8 @@ impl MatchOptions {
472472
pub trait MatchConfig: Send + Sync + 'static {
473473
fn options() -> MatchOptions;
474474

475-
/// The typical number of tokens in a value this index stores: 20 for a
476-
/// name column, 200 for a notes column. A prior, used to size a query's
475+
/// The typical number of distinct tokens in a value this index stores:
476+
/// 20 for a name column, 200 for a notes column. A prior, used to size a query's
477477
/// fetch ([`Stats::declared`](crate::op::Stats::declared)) before the
478478
/// column's filters have been measured. It is not part of the index's
479479
/// identity and changes no term. `None`, the default, declares no

‎packages/stack-encrypt/tests/op.rs‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -743,12 +743,22 @@ async fn target_limit_without_a_prior_needs_a_measured_column() {
743743
.entry(stored.terms.1.positions().len() as u32)
744744
.or_default() += 1;
745745
}
746-
let stats = Stats::measured(3, histogram);
747-
let Limit::Rows(rows) = query.target_limit(1, &stats) else {
746+
// The pass rate the histogram gives this query, over a large table.
747+
let bits = query.positions().len() as i32;
748+
let rows_seen: u64 = histogram.values().sum();
749+
let p = histogram
750+
.iter()
751+
.map(|(&set, &n)| n as f64 * (f64::from(set) / 256.0).powi(bits))
752+
.sum::<f64>()
753+
/ rows_seen as f64;
754+
let lambda = p * 1_000_000.0;
755+
let stats = Stats::measured(1_000_000, histogram);
756+
let Limit::Rows(rows) = query.target_limit(10, &stats) else {
748757
panic!("a measured column gives a finite fetch");
749758
};
759+
let extra = (rows - 10) as f64;
750760
assert!(
751-
(1..=4).contains(&rows),
752-
"1 wanted, at most every row: {rows}"
761+
extra >= lambda && extra <= lambda + 4.0 * lambda.sqrt() + 3.0,
762+
"10 wanted, λ = {lambda}: {rows}"
753763
);
754764
}

0 commit comments

Comments
 (0)