You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5db609d
Browse filesBrowse the repository at this point in the historyBrowse files
feat(stack-encrypt): size a match query's fetch with target_limit
ADR-0009 decision 5: a caller who wants `want` real matches from a
probabilistic query fetches `want + Poisson_quantile(λ = p·N, confidence)`
rows, where `p` is the mean over stored filters of `f^b`, `f` a filter's
fill and `b` the bits the query sets.
- `Probabilistic::target_limit(&self, want, &Stats) -> Limit`.
- `op::Stats::declared(rows)` uses a new `MatchConfig::expected_tokens()`
prior, a default method returning `None`, so existing configurations
compile unchanged; it is not part of the index's identity.
- `op::Stats::measured(rows, histogram)` takes `(set bits, rows)` buckets
from one sampled aggregate over the column, and takes the mean of the
power, never the power of the mean fill. An empty histogram falls back
to the prior.
- `op::Limit` is `Rows(n)` or `Unbounded`, the latter when nothing is known
of the stored filters or the confidence is not in (0, 1). `fetch_or(page)`
reads either as a fetch size.
- The quantile sums the Poisson mass exactly in log space from
`λ − 10√λ`, so it costs about `12√λ` terms; past `λ = 1e7` it is the
conservative `λ + 8√λ`.
Unit tests reproduce the ADR's worked table and its mixed-length column,
and pin the quantile to reference values computed with `lgamma`.
Closes#1184
Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
Copy file name to clipboardExpand all lines: packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md
+22-13Lines changed: 22 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -344,9 +344,10 @@ takes them as numbers in `Stats` and does not care where they came from.
344
344
There are two sources, and a column has the first one from the day it is
345
345
declared.
346
346
347
-
**Declared.** A match index declares `expected_tokens`, the typical token
348
-
count of the values it will hold, beside `m` and `k`: 20 for a name column,
349
-
200 for a notes column. From it `Stats::declared(rows)` takes
347
+
**Declared.** A match configuration declares `expected_tokens()`, the typical
348
+
token count of the values it will hold, beside the `m` and `k` of its
349
+
options: 20 for a name column, 200 for a notes column. It is not part of the
350
+
index's identity and changes no term. From it `Stats::declared(rows)` takes
350
351
`f = 1 − e^(−k·expected_tokens/m)` and `p = f^b`. This is a prior, and
351
352
`rows` is a planning estimate too. It is enough to warn before a query runs
352
353
(a three-letter needle against a notes column) and to size a first fetch; it
@@ -369,8 +370,16 @@ query has. The sample bounds the cost on a large table; `TABLESAMPLE` is a
369
370
page sample, which is fine for a distribution and unsuited to an exact count.
370
371
It is also a sample of pages, so on a table of a few thousand rows a 1%
371
372
sample can select none; below that size the clause is dropped and the whole
372
-
table is read, which costs nothing worth saving. `rows` comes from `SELECT reltuples FROM pg_class WHERE oid =
373
-
'users'::regclass`, a catalog read.
373
+
table is read, which costs nothing worth saving. `rows` comes from
374
+
`SELECT reltuples FROM pg_class WHERE oid = 'users'::regclass`, a catalog
375
+
read. An empty histogram falls back to the declared prior.
376
+
377
+
`target_limit` returns a `Limit`: `Rows(n)`, or `Unbounded` when the
378
+
statistics say nothing about the stored filters (no prior and no
379
+
histogram) or the confidence is not below 1. `fetch_or(page)` reads either
380
+
as a fetch size, for a caller that pages anyway. Past ten million expected
381
+
false positives the quantile is the conservative `λ + 8√λ`, since the query
382
+
is hopeless at that point.
374
383
375
384
Where the numbers live and who keeps them in step was asked on the review,
376
385
so it is written down. They are read from the server by the application, at
@@ -431,7 +440,7 @@ shot, with a bound, and the completeness check from decision 5:
0 commit comments