Skip to content

Commit aa91861

Browse files
committed
feat(stack-encrypt): verify a match query's rows after decrypt
A Bloom filter can hold every bit of a query without the value holding the needle, so a `Matches` query returns false positives. ADR-0009 decision 4 makes such a term `Probabilistic`, with `verify` as the operator's own meaning run on a decrypted value. - `op::Probabilistic { type Plaintext; fn verify(&self, &Plaintext) -> bool }`. Infallible: every outcome other than a match is `false`. - `MatchQuery` keeps the needle's tokens as the index cut them, through a new `Supports::carry` taken before the value is consumed. `verify` cuts the decrypted value the same way and checks every needle token is among its tokens. The tokens are zeroized on drop and never shown by `Debug`. - `Supports::Query` is bounded by `op::TermFor<Op>`: any query term answers `Eq`, only a probabilistic one answers `Matches`, so an index cannot claim a text match without saying how its answer is checked (tests/ui/op_inexact_operator_needs_probabilistic.rs). - `verify` on an exact term does not compile (tests/ui/op_verify_exact_term.rs). The tests plant a real Bloom false positive under a 32-bit filter and check `verify` drops it. Closes #1183 Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
1 parent b6b8d9d commit aa91861

10 files changed

Lines changed: 350 additions & 43 deletions

‎packages/stack-encrypt/CHANGELOG.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3636
- **`Match` is a value.** `.with((Equality, Match))` declares the match index
3737
under the default configuration, as `Equality` declares equality;
3838
`Match::<MyConfig>::new()` is another configuration.
39+
- **`verify` drops a match query's false positives after decrypt.** A
40+
`MatchQuery` is `op::Probabilistic`: `query.verify(&value)` is `true` when
41+
every token of the query's value, cut by the index's own tokenizer and
42+
case folding, is a token of the decrypted value. Filter decrypted rows
43+
with it. The query keeps its value's tokens to do this; they are zeroized on drop
44+
and never shown by `Debug`. An exact term has no `verify`, and an index
45+
can answer `op::Matches` only with a probabilistic term (`op::TermFor`).
3946

4047
- **Errors now expose stable codes, help text, and structured details.**
4148
All error types implement `miette::Diagnostic` with codes such as
@@ -85,9 +92,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
8592
`QueryBuilder`, `QueryVia`, `QueryIndexBuilder`, `QueryIndex` and
8693
`QueryOp`, and the plan shapes implement `plan::Chooses` and
8794
`plan::Resolves`.
88-
- **`Match::default()` is `Match`.** The match index under the default
95+
- **`Match` is `Match`.** The match index under the default
8996
configuration is the value `Match`, and `Default` is now implemented for
90-
every `Match<O>`, so `Match::default()` no longer infers the configuration.
97+
every `Match<O>`, so `Match` no longer infers the configuration.
9198

9299
- **Update pattern matches for `dynamic::Error`.** `Context`, `Plan`,
93100
`Source`, and `Record` now carry

‎packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -144,9 +144,11 @@ query against a case-folding index is a type error, not an empty result.
144144
pub trait Operator: Copy + 'static {}
145145

146146
pub trait Supports<Op: Operator, S>: Index<S> {
147-
type Query: QueryTerm; // what this index sends for this operator
147+
type Query: TermFor<Op>; // what this index sends for `Op`
148+
type Carry; // what the query keeps of the value
148149
fn check(&self) -> Result<(), PlanError>;
149-
fn query(term: Self::Term, token: QueryToken) -> Self::Query;
150+
fn carry(&self, value: &S) -> Self::Carry;
151+
fn query(carry: Self::Carry, term: Self::Term, token: QueryToken) -> Self::Query;
150152
}
151153
```
152154

@@ -164,10 +166,14 @@ ciphertext; its `Term` is left and right. A Bloom index's
164166
`Supports<Matches, S>::Query` is a filter over the needle (`MatchQuery`); its
165167
`Term` is a filter over the value (`MatchTerms`).
166168

167-
The bound tightens per operator. `Supports<Matches, S>` requires
168-
`Query: Probabilistic`; `Supports<Nearest, S>` requires `Query: Ranked`. An
169-
index cannot claim an inexact operator without saying how its answer is
170-
checked.
169+
The bound tightens per operator, through `TermFor<Op>`: any `QueryTerm`
170+
answers `Eq`, but only a `Probabilistic` one answers `Matches`, and only a
171+
`Ranked` one will answer `Nearest`. An index cannot claim an inexact
172+
operator without saying how its answer is checked.
173+
174+
`carry` takes what the query term keeps of the value before the value is
175+
consumed: nothing for an exact term, the query's tokens for a match query,
176+
which `verify` needs and the filter cannot give back.
171177

172178
### 3. The domain type chooses the best index for an operator
173179

‎packages/stack-encrypt/src/op.rs‎

Lines changed: 70 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,9 @@
5555
use std::fmt;
5656
use std::marker::PhantomData;
5757

58-
use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms};
58+
use stack_kms::MaybeSend;
59+
60+
use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens};
5961
use crate::target::{Equality, Index, Match};
6062
use crate::PlanError;
6163

@@ -199,29 +201,78 @@ pub trait QueryTerm {}
199201
impl QueryTerm for EqualityTerm {}
200202
impl<O> QueryTerm for MatchQuery<O> {}
201203

204+
/// A query term whose answer can hold **false positives and no false
205+
/// negatives**: a row the server returns may not match, and a row that
206+
/// matches is always returned.
207+
///
208+
/// [`verify`](Self::verify) is the operator's own meaning, run on a
209+
/// decrypted value: it decides whether a row the server returned is a real
210+
/// match. The library never runs a query, so it cannot drop the false
211+
/// positives itself; a caller filters the decrypted rows with `verify`.
212+
///
213+
/// An exact term does not implement this, so `verify` on one is a compile
214+
/// error rather than a method that is always `true`.
215+
pub trait Probabilistic: QueryTerm {
216+
/// The decrypted value `verify` reads: the plaintext the index was
217+
/// declared over, which is not always the value the user wrote.
218+
type Plaintext: ?Sized;
219+
220+
/// Whether `value`, a row's decrypted value, really matches the query.
221+
///
222+
/// Infallible: it runs on a value already decrypted and already the type
223+
/// the index was declared over, so every outcome other than a match is
224+
/// `false`, never an error.
225+
fn verify(&self, value: &Self::Plaintext) -> bool;
226+
}
227+
228+
/// The query terms that may answer the operator `Op`: what
229+
/// [`Supports::Query`] must be.
230+
///
231+
/// Most operators take any [`QueryTerm`]. One whose answer is inherently
232+
/// approximate requires more: a term that answers [`Matches`](struct@Matches) must be
233+
/// [`Probabilistic`], so an index cannot claim a text match without saying
234+
/// how its answer is checked. An operator defined in another crate states
235+
/// its own requirement by implementing this.
236+
#[diagnostic::on_unimplemented(
237+
message = "`{Self}` cannot answer `{Op}`",
238+
label = "this query term does not meet the operator's requirement",
239+
note = "a term that answers `op::Matches` must implement `op::Probabilistic`"
240+
)]
241+
pub trait TermFor<Op: Operator>: QueryTerm {}
242+
243+
impl<T: QueryTerm> TermFor<Eq> for T {}
244+
impl<C: Case, T: Probabilistic> TermFor<Matches<C>> for T {}
245+
202246
/// An index that answers the operator `Op` over plaintexts `S`, and the
203247
/// query term it sends for it.
204248
///
205249
/// [`Index::Term`] is what the index derives; [`query`](Self::query) makes
206-
/// the query term from it. For an equality index the two are the same
207-
/// HMAC. For a match index they are not: the query is a filter over the
208-
/// query's value, and its type says so.
250+
/// the query term from it and from what [`carry`](Self::carry) kept of the
251+
/// query's value. For an equality index the two are the same HMAC and
252+
/// nothing is kept. For a match index they are not: the query is a filter
253+
/// over the query's value, and it keeps that value's tokens so it can
254+
/// [`verify`](Probabilistic::verify) a row.
209255
#[diagnostic::on_unimplemented(
210256
message = "`{Self}` does not answer `{Op}` over `{S}`",
211257
label = "this index cannot answer the operator",
212258
note = "an equality index answers `op::Eq`; a match index answers `op::Matches` over text, in the case its configuration fixes"
213259
)]
214260
pub trait Supports<Op: Operator, S>: Index<S> {
215261
/// What this index sends for `Op`.
216-
type Query: QueryTerm;
262+
type Query: TermFor<Op>;
263+
/// What the query term keeps of the query's value, taken before the
264+
/// value is consumed: `()` for an exact term.
265+
type Carry: MaybeSend + 'static;
217266
/// Why this index cannot answer `Op` as configured, if it cannot: a
218267
/// match index whose [`MatchCase`] disagrees with its options.
219268
fn check(&self) -> Result<(), PlanError>;
220-
/// The query term, from the term the index derived for the query's
221-
/// value. Only a query chain calls this: `QueryToken` cannot be made
222-
/// outside this crate, so a stored term cannot be passed off as a
223-
/// query.
224-
fn query(term: Self::Term, token: QueryToken) -> Self::Query;
269+
/// Take what the query term keeps of `value`.
270+
fn carry(&self, value: &S) -> Self::Carry;
271+
/// The query term, from what was kept and the term the index derived
272+
/// for the query's value. Only a query chain calls this: `QueryToken`
273+
/// cannot be made outside this crate, so no caller can pair one query's tokens
274+
/// with another value's filter, or pass a stored term off as a query.
275+
fn query(carry: Self::Carry, term: Self::Term, token: QueryToken) -> Self::Query;
225276
}
226277

227278
/// Proof that [`Supports::query`] is called by a query chain. Only this
@@ -238,10 +289,12 @@ impl QueryToken {
238289

239290
impl<S: vitaminc_prf::PrfValue> Supports<Eq, S> for Equality {
240291
type Query = EqualityTerm;
292+
type Carry = ();
241293
fn check(&self) -> Result<(), PlanError> {
242294
Ok(())
243295
}
244-
fn query(term: EqualityTerm, _: QueryToken) -> EqualityTerm {
296+
fn carry(&self, _: &S) {}
297+
fn query((): (), term: EqualityTerm, _: QueryToken) -> EqualityTerm {
245298
term
246299
}
247300
}
@@ -252,6 +305,7 @@ where
252305
O: MatchCase,
253306
{
254307
type Query = MatchQuery<O>;
308+
type Carry = QueryTokens;
255309
fn check(&self) -> Result<(), PlanError> {
256310
let folds = O::options().downcase;
257311
if folds == O::Case::FOLDS {
@@ -263,8 +317,11 @@ where
263317
})
264318
}
265319
}
266-
fn query(term: MatchTerms<O>, _: QueryToken) -> MatchQuery<O> {
267-
MatchQuery::from_terms(term)
320+
fn carry(&self, value: &S) -> QueryTokens {
321+
QueryTokens::of::<O>(value.as_ref())
322+
}
323+
fn query(tokens: QueryTokens, term: MatchTerms<O>, _: QueryToken) -> MatchQuery<O> {
324+
MatchQuery::new(term, tokens)
268325
}
269326
}
270327

‎packages/stack-encrypt/src/plan/chain.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -980,9 +980,10 @@ where
980980
where
981981
'a: 'p,
982982
{
983+
let carry = self.inner.index.carry(&self.inner.value);
983984
self.inner
984985
.prepare(keyset, scoped)
985-
.map(|term| I::query(term, QueryToken::new()))
986+
.map(move |term| I::query(carry, term, QueryToken::new()))
986987
}
987988
}
988989

‎packages/stack-encrypt/src/sem/mod.rs‎

Lines changed: 72 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,7 @@ mod tokenize;
128128

129129
pub use tokenize::Tokenizer;
130130

131+
use std::collections::HashSet;
131132
use std::fmt;
132133
use std::marker::PhantomData;
133134

@@ -142,7 +143,7 @@ use vitaminc_prf::{
142143
PrfVisitorError, SeqAccess,
143144
};
144145
use vitaminc_protected::NonEmpty;
145-
use zeroize::Zeroize;
146+
use zeroize::{Zeroize, Zeroizing};
146147

147148
use stack_kms::MaybeSend;
148149

@@ -652,14 +653,24 @@ impl<O> Eq for MatchTerms<O> {}
652653
/// stored filter is never sent as a query. ADR-0009 records why. A
653654
/// `MatchQuery` comes from a query chain ([`op::Matches`](struct@crate::op::Matches)),
654655
/// never from a target.
656+
///
657+
/// A Bloom filter can hold every bit of the query without the value holding
658+
/// the query's text, so a match query is
659+
/// [`Probabilistic`](crate::op::Probabilistic): it keeps the query's tokens,
660+
/// as the index cut them, and [`verify`](crate::op::Probabilistic::verify)
661+
/// checks a decrypted value against them. The tokens are plaintext: they
662+
/// never leave the process, never appear in `Debug`, and are zeroized when
663+
/// the query is dropped.
655664
pub struct MatchQuery<O = DefaultMatch> {
656665
terms: MatchTerms<O>,
666+
tokens: QueryTokens,
657667
}
658668

659669
impl<O> MatchQuery<O> {
660-
/// The query term of the filter `terms`, derived over the query's value.
661-
pub(crate) fn from_terms(terms: MatchTerms<O>) -> Self {
662-
Self { terms }
670+
/// The query term of the filter `terms`, derived over the value whose
671+
/// tokens are `tokens`.
672+
pub(crate) fn new(terms: MatchTerms<O>, tokens: QueryTokens) -> Self {
673+
Self { terms, tokens }
663674
}
664675

665676
/// The set positions of the query's filter, sorted ascending, no
@@ -679,22 +690,47 @@ impl<O> MatchQuery<O> {
679690
}
680691
}
681692

693+
/// A row is a real match when every token of the query is a token of the
694+
/// value, cut by the index's own tokenizer and case folding. That is the
695+
/// meaning a Bloom match approximates, so `verify` keeps every row the
696+
/// filter would have kept had it no false positives: order and repetition
697+
/// do not count, and a match is not a substring test.
698+
impl<O: MatchConfig> crate::op::Probabilistic for MatchQuery<O> {
699+
type Plaintext = str;
700+
701+
fn verify(&self, value: &str) -> bool {
702+
let options = O::options();
703+
let tokens = Zeroizing::new(tokenize::tokenize(
704+
value,
705+
options.tokenizer,
706+
options.downcase,
707+
));
708+
let have: HashSet<&str> = tokens.iter().map(String::as_str).collect();
709+
!self.tokens.0.is_empty() && self.tokens.0.iter().all(|t| have.contains(t.as_str()))
710+
}
711+
}
712+
713+
/// Positions only: the tokens are plaintext.
682714
impl<O> fmt::Debug for MatchQuery<O> {
683715
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
684716
f.debug_struct("MatchQuery")
685717
.field("positions", &self.terms.positions)
686-
.finish()
718+
.finish_non_exhaustive()
687719
}
688720
}
689721

690722
impl<O> Clone for MatchQuery<O> {
691723
fn clone(&self) -> Self {
692724
Self {
693725
terms: self.terms.clone(),
726+
tokens: self.tokens.clone(),
694727
}
695728
}
696729
}
697730

731+
/// Two queries are equal when they send the same filter. Under one key and
732+
/// context the filter is a function of the query's tokens, so this never
733+
/// compares the plaintext.
698734
impl<O> PartialEq for MatchQuery<O> {
699735
fn eq(&self, other: &Self) -> bool {
700736
self.terms == other.terms
@@ -703,6 +739,24 @@ impl<O> PartialEq for MatchQuery<O> {
703739

704740
impl<O> Eq for MatchQuery<O> {}
705741

742+
/// The query's tokens, as a match index cuts them from the query's value:
743+
/// what a [`MatchQuery`] keeps to verify a row. Plaintext, so it is
744+
/// zeroized on drop and has no `Debug`. Built only by a query chain.
745+
#[derive(Clone)]
746+
pub struct QueryTokens(Zeroizing<Vec<String>>);
747+
748+
impl QueryTokens {
749+
/// The tokens of `text` under `O`'s tokenizer and case folding.
750+
pub(crate) fn of<O: MatchConfig>(text: &str) -> Self {
751+
let options = O::options();
752+
Self(Zeroizing::new(tokenize::tokenize(
753+
text,
754+
options.tokenizer,
755+
options.downcase,
756+
)))
757+
}
758+
}
759+
706760
/// A [`PrfVisitor`] that folds a sequence of per-token PRF blocks into
707761
/// Bloom-filter bit positions: `k` little-endian 2-byte slices of each block,
708762
/// masked to the filter size.
@@ -1277,6 +1331,19 @@ mod tests {
12771331

12781332
use super::*;
12791333

1334+
#[test]
1335+
fn a_match_query_with_no_tokens_verifies_nothing() {
1336+
// A query chain refuses token-less text before a term exists, so
1337+
// this cannot be built through the API; the guard is for a query
1338+
// built any other way, which must never match every row.
1339+
let query = MatchQuery::<DefaultMatch>::new(
1340+
MatchTerms::normalised(vec![1, 2, 3]),
1341+
QueryTokens(Zeroizing::new(Vec::new())),
1342+
);
1343+
assert!(!crate::op::Probabilistic::verify(&query, "anything at all"));
1344+
assert!(!crate::op::Probabilistic::verify(&query, ""));
1345+
}
1346+
12801347
/// The Bloom fold reads a token *stream*: handed a map-shaped PRF input
12811348
/// it refuses, rather than folding the entries' blocks as if they were
12821349
/// tokens.

0 commit comments

Comments
 (0)