Skip to content

Commit 936f345

Browse files
committed
fix(stack-encrypt): address the review of match verify
- `Supports::query` takes the sealed `QueryToken` (from the #1190 fix), so no caller can pair a needle with another value's filter and have `verify` drop the real rows. `Needle`'s doc says how one is joined. - `TermFor`'s doc and ADR-0009 say the "only a probabilistic term answers `Matches`" rule binds this crate's terms; a term in another crate can opt in by hand. - The tokenizer zeroizes its own copies of the text: the folded string, its characters, and each n-gram, built at its final size so it does not reallocate. `verify` runs it on every decrypted row. - `MatchQuery`'s `PartialEq` says two needles can share a filter and still verify differently. - ADR-0009 decision 4's code shows the trait as this PR ships it; `target_limit` joins it with #1184. - Tests: `verify` is a token test, not a substring test ("abcd" verifies "abc bcd"), and the standard tokenizer verifies whole words in any order. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
1 parent aa91861 commit 936f345

6 files changed

Lines changed: 105 additions & 21 deletions

File tree

‎packages/stack-encrypt/CHANGELOG.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4040
`MatchQuery` is `op::Probabilistic`: `query.verify(&value)` is `true` when
4141
every token of the query's value, cut by the index's own tokenizer and
4242
case folding, is a token of the decrypted value. Filter decrypted rows
43-
with it. The query keeps its value's tokens to do this; they are zeroized on drop
44-
and never shown by `Debug`. An exact term has no `verify`, and an index
45-
can answer `op::Matches` only with a probabilistic term (`op::TermFor`).
43+
with it. The query keeps its value's tokens to do this; they are zeroized
44+
on drop and never shown by `Debug`, and the tokenizer now zeroizes its own
45+
copies of the text. An exact term has no `verify`, and none of this
46+
crate's indexes can answer `op::Matches` but with a probabilistic term
47+
(`op::TermFor`).
4648

4749
- **Errors now expose stable codes, help text, and structured details.**
4850
All error types implement `miette::Diagnostic` with codes such as

‎packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -168,8 +168,10 @@ ciphertext; its `Term` is left and right. A Bloom index's
168168

169169
The bound tightens per operator, through `TermFor<Op>`: any `QueryTerm`
170170
answers `Eq`, but only a `Probabilistic` one answers `Matches`, and only a
171-
`Ranked` one will answer `Nearest`. An index cannot claim an inexact
172-
operator without saying how its answer is checked.
171+
`Ranked` one will answer `Nearest`. None of this crate's indexes can claim
172+
an inexact operator without saying how its answer is checked. The rule binds
173+
this crate's terms through blanket impls; a term in another crate can opt
174+
into `TermFor` by hand, so there it is a guide rather than a proof.
173175

174176
`carry` takes what the query term keeps of the value before the value is
175177
consumed: nothing for an exact term, the query's tokens for a match query,
@@ -215,7 +217,8 @@ pub trait QueryTerm {}
215217
pub trait Probabilistic: QueryTerm {
216218
type Plaintext;
217219
fn verify(&self, value: &Self::Plaintext) -> bool;
218-
fn target_limit(&self, want: u64, stats: &Stats) -> Limit;
220+
// `target_limit(&self, want, &Stats) -> Limit` joins it with decision 5
221+
// (#1184).
219222
}
220223

221224
pub trait Ranked: QueryTerm {

‎packages/stack-encrypt/src/op.rs‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -230,9 +230,14 @@ pub trait Probabilistic: QueryTerm {
230230
///
231231
/// Most operators take any [`QueryTerm`]. One whose answer is inherently
232232
/// approximate requires more: a term that answers [`Matches`](struct@Matches) must be
233-
/// [`Probabilistic`], so an index cannot claim a text match without saying
234-
/// how its answer is checked. An operator defined in another crate states
235-
/// its own requirement by implementing this.
233+
/// [`Probabilistic`], so none of this crate's indexes can claim a text
234+
/// match without saying how its answer is checked.
235+
///
236+
/// The rule binds this crate's terms, through the blanket impls below. A
237+
/// term defined in another crate can implement this for an operator by
238+
/// hand, and coherence lets it; the bound is a guide there, not a proof. An
239+
/// operator defined in another crate states its requirement the same way,
240+
/// one term type at a time.
236241
#[diagnostic::on_unimplemented(
237242
message = "`{Self}` cannot answer `{Op}`",
238243
label = "this query term does not meet the operator's requirement",

‎packages/stack-encrypt/src/sem/mod.rs‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -728,9 +728,11 @@ impl<O> Clone for MatchQuery<O> {
728728
}
729729
}
730730

731-
/// Two queries are equal when they send the same filter. Under one key and
732-
/// context the filter is a function of the query's tokens, so this never
733-
/// compares the plaintext.
731+
/// Two queries are equal when they send the same filter. This compares no
732+
/// plaintext. Two query values can share a filter by chance, as a short
733+
/// value under a small filter does, and those queries are equal here yet
734+
/// verify differently: equal queries select the same rows, not the same
735+
/// matches.
734736
impl<O> PartialEq for MatchQuery<O> {
735737
fn eq(&self, other: &Self) -> bool {
736738
self.terms == other.terms
@@ -741,7 +743,12 @@ impl<O> Eq for MatchQuery<O> {}
741743

742744
/// The query's tokens, as a match index cuts them from the query's value:
743745
/// what a [`MatchQuery`] keeps to verify a row. Plaintext, so it is
744-
/// zeroized on drop and has no `Debug`. Built only by a query chain.
746+
/// zeroized on drop and has no `Debug`.
747+
/// [`Supports::carry`](crate::op::Supports::carry) makes one; only a query
748+
/// chain can join it to a filter
749+
/// ([`Supports::query`](crate::op::Supports::query) takes a token no caller
750+
/// can make), so one query's tokens never verify rows for another value's
751+
/// filter.
745752
#[derive(Clone)]
746753
pub struct QueryTokens(Zeroizing<Vec<String>>);
747754

@@ -1334,8 +1341,8 @@ mod tests {
13341341
#[test]
13351342
fn a_match_query_with_no_tokens_verifies_nothing() {
13361343
// A query chain refuses token-less text before a term exists, so
1337-
// this cannot be built through the API; the guard is for a query
1338-
// built any other way, which must never match every row.
1344+
// this cannot reach a `MatchQuery` through the API; the guard is for
1345+
// one built any other way, which must never match every row.
13391346
let query = MatchQuery::<DefaultMatch>::new(
13401347
MatchTerms::normalised(vec![1, 2, 3]),
13411348
QueryTokens(Zeroizing::new(Vec::new())),

‎packages/stack-encrypt/src/sem/tokenize.rs‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
//! is that empty tokens are dropped (v1 keeps the empty strings its separator
1111
//! split produces, which only add noise bits to every filter).
1212
13+
use zeroize::Zeroizing;
14+
1315
/// How text is split into tokens before each token is run through the PRF.
1416
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
1517
pub enum Tokenizer {
@@ -45,12 +47,17 @@ impl Default for Tokenizer {
4547
/// term can never reach a query.
4648
///
4749
/// [`match_terms`]: crate::StackCipher::match_terms
50+
///
51+
/// The text is plaintext, so every copy made on the way is zeroized when it
52+
/// is dropped: the folded text, its characters, and each n-gram, which is
53+
/// built at its final size so it never reallocates. The tokens returned are
54+
/// the caller's to zeroize.
4855
pub(crate) fn tokenize(text: &str, tokenizer: Tokenizer, downcase: bool) -> Vec<String> {
49-
let text = if downcase {
56+
let text = Zeroizing::new(if downcase {
5057
text.to_lowercase()
5158
} else {
5259
text.to_string()
53-
};
60+
});
5461

5562
match tokenizer {
5663
Tokenizer::Standard => text
@@ -59,19 +66,27 @@ pub(crate) fn tokenize(text: &str, tokenizer: Tokenizer, downcase: bool) -> Vec<
5966
.map(str::to_string)
6067
.collect(),
6168
Tokenizer::Ngram { length } => {
62-
let chars: Vec<char> = text.chars().collect();
69+
let chars: Zeroizing<Vec<char>> = Zeroizing::new(text.chars().collect());
6370
// As in the v1 indexer's `process_ngram`: shorter than one gram
6471
// means no tokens (never a partial or whole-text token, which
6572
// could not match any stored gram).
6673
if chars.len() < length {
6774
Vec::new()
6875
} else {
69-
chars.windows(length).map(|w| w.iter().collect()).collect()
76+
chars.windows(length).map(gram).collect()
7077
}
7178
}
7279
}
7380
}
7481

82+
/// One n-gram as a `String` allocated at its final size, so building it
83+
/// leaves no reallocated copy behind.
84+
fn gram(chars: &[char]) -> String {
85+
let mut gram = String::with_capacity(chars.iter().map(|c| c.len_utf8()).sum());
86+
gram.extend(chars);
87+
gram
88+
}
89+
7590
#[cfg(test)]
7691
mod tests {
7792
use super::*;

‎packages/stack-encrypt/tests/op.rs‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ async fn a_matches_query_is_a_filter_over_the_query_value_that_the_stored_filter
133133
}
134134

135135
#[tokio::test]
136-
async fn a_match_index_with_its_own_configuration_is_named_with_op_via() {
136+
async fn a_match_index_with_its_own_configuration_is_named_with_via() {
137137
let cipher = stack_cipher().await;
138138
let plan = people_plan();
139139
let mut row = cipher.encrypt(&person()).using(&plan).await.unwrap();
@@ -267,7 +267,7 @@ async fn a_target_answers_an_operator_through_the_index_it_names() {
267267
}
268268

269269
#[tokio::test]
270-
async fn a_target_with_several_terms_names_the_index_with_op_via() {
270+
async fn a_target_with_several_terms_names_the_index_with_via() {
271271
let cipher = stack_cipher().await;
272272
let plan = Plan::value::<String>()
273273
.context("users/email")
@@ -572,3 +572,55 @@ async fn a_match_query_debugs_without_its_tokens() {
572572
assert!(!shown.contains(token), "{shown}");
573573
}
574574
}
575+
576+
/// Whole words, as the v1 match indexer's standard tokenizer splits them.
577+
struct Words;
578+
impl MatchConfig for Words {
579+
fn options() -> MatchOptions {
580+
MatchOptions {
581+
tokenizer: stack_encrypt::sem::Tokenizer::Standard,
582+
..Default::default()
583+
}
584+
}
585+
}
586+
impl MatchCase for Words {
587+
type Case = op::CaseInsensitive;
588+
}
589+
590+
/// A Bloom match asks for the query's tokens, in any order and anywhere;
591+
/// it is not a substring test, and `verify` must not become one, or it
592+
/// would drop rows the index rightly returned.
593+
#[tokio::test]
594+
async fn verify_is_a_token_test_not_a_substring_test() {
595+
let cipher = stack_cipher().await;
596+
let plan = people_plan();
597+
let query = cipher
598+
.query(op::Matches, "abcd")
599+
.on(&plan.field("email").unwrap())
600+
.await
601+
.unwrap();
602+
assert!(query.verify("abc bcd"), "both 3-grams, never adjacent");
603+
assert!(!query.verify("abc"), "one 3-gram of two");
604+
605+
let words = Plan::context("notes/body")
606+
.with::<String, _>(Match::<Words>::new())
607+
.build()
608+
.unwrap();
609+
let stored: Encrypted<MatchTerms<Words>> = cipher
610+
.encrypt(&"Hello, big World!".to_owned())
611+
.using(&words)
612+
.await
613+
.unwrap();
614+
let query = cipher
615+
.query(op::Matches, "world hello")
616+
.via(Match::<Words>::new())
617+
.on(&words)
618+
.await
619+
.unwrap();
620+
assert!(stored.terms.matches(&query));
621+
assert!(query.verify("Hello, big World!"));
622+
assert!(
623+
!query.verify("worldly hello"),
624+
"a word is a token, not a prefix"
625+
);
626+
}

0 commit comments

Comments
 (0)