Skip to content

Commit be7e1e3

Browse files
committed
fix(eql-bindings, stashgen): document TextMatch's limits, and teach the test engine TextMatch
- The eql-bindings README, the stashgen README and the changeset say that only `stored @@ query` matches until #1171 is fixed, and that a TextMatchQuery for text shorter than three characters is refused too. - The PostgreSQL test's reversed-order assertion names #1171, and the filter test says its negative case is a property of one key source and configuration, not of Bloom filters. - enginetest.Static produces TextMatch; the refusal case for an EQL type the engine cannot produce now names IntegerEq; the users golden case stores Name as TextMatch, so generation of its field methods is exercised. - encryption_gap drops its Bloom branch, which no catalog domain reached once text_match became producible. - The generated TextMatchQuery doc keeps the plaintext line without the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
1 parent 92e7d33 commit be7e1e3

11 files changed

Lines changed: 48 additions & 26 deletions

File tree

‎.changeset/eql-v4-bindings.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
'@cipherstash/eql': minor
33
---
44

5-
**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused), from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.
5+
**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.
66

77
Nothing a released version shipped has changed. The v3 types, the v3 SQL and the TypeScript package are as they were; the `stack-encrypt` derives on the v3 text types and the `v3::targets` module were added after the last release and are moved to v4 here, not removed from a release.

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,8 @@ The engine produces these EQL types today, each from one Go field type:
161161
| `Double` | `float64` |
162162

163163
The width must match: an `int` or `int64` field is not an `Integer`, and a `float64` field is not a `Real`.
164-
A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index.
164+
A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too.
165+
Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed.
165166
`encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet.
166167
Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`.
167168

‎languages/golang/stashgen/enginetest/enginetest.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ var eqlTypes = []struct {
3232
{stashgen.EQLType{Name: "Bigint", Plaintext: stashgen.KindInt, Producible: true}, "int64"},
3333
{stashgen.EQLType{Name: "Text", Plaintext: stashgen.KindString, Producible: true}, ""},
3434
{stashgen.EQLType{Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true}, ""},
35+
{stashgen.EQLType{Name: "TextMatch", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexMatch}, Query: "TextMatchQuery", Producible: true}, ""},
3536
{stashgen.EQLType{Name: "Boolean", Plaintext: stashgen.KindBool, Producible: true}, ""},
3637
{stashgen.EQLType{Name: "Real", Plaintext: stashgen.KindFloat, Producible: true}, "float32"},
3738
{stashgen.EQLType{Name: "Double", Plaintext: stashgen.KindFloat, Producible: true}, "float64"},

‎languages/golang/stashgen/refusal_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ func TestRefusals(t *testing.T) {
9191
{"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"},
9292
{"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"},
9393
{"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"},
94-
{"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "has no EQL type TextMatch"},
94+
{"an EQL type the engine cannot produce yet", user(ctx + "\tAge int32 `stash:\"age,encrypt_into=IntegerEq\"`"), stashgen.Request{Type: "User"}, "Age", "has no EQL type IntegerEq"},
9595
{"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"},
9696
{"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"},
9797
{"an index option a query term cannot use", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "a query term uses only the default index options"},

‎languages/golang/stashgen/testdata/cases/users/model.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,6 @@ type User struct {
88
_ struct{} `stash:"context=users"`
99
ID int64 `stash:"id,passthrough" db:"id" gorm:"primaryKey"`
1010
Email string `stash:"email,encrypt_into=TextEq" db:"email"`
11-
Name string `stash:"name,encrypt_into=TextEq" db:"name"`
11+
Name string `stash:"name,encrypt_into=TextMatch" db:"name"`
1212
Internal string `stash:"-"`
1313
}

‎languages/golang/stashgen/testdata/cases/users/user_stash.go.golden‎

Lines changed: 9 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,15 @@
1-
// Package eql is a TEST STUB of the generated EQL types: the one type the
2-
// engine produces today, and its query type.
1+
// Package eql is a TEST STUB of the generated EQL types: the types the
2+
// golden cases name, and their query types.
33
package eql
44

55
// TextEq is an EQL value with an equality term.
66
type TextEq []byte
77

88
// TextEqQuery is the query value for TextEq.
99
type TextEqQuery []byte
10+
11+
// TextMatch is an EQL value with a match term.
12+
type TextMatch []byte
13+
14+
// TextMatchQuery is the query value for TextMatch.
15+
type TextMatchQuery []byte

‎packages/eql/crates/eql-bindings/README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -129,9 +129,11 @@ vitaminc 0.5's per-width domains would stop matching after that upgrade.
129129
stored values and queries alike. The configuration is part of the stored
130130
format, so changing it means a new domain or payload version. A filter can
131131
match a row that does not contain the query, so re-check after decrypting
132-
where that matters. Text too short to make a 3-gram (empty strings, two-letter
133-
codes) cannot be stored in a `TextMatch` column yet: Stack Encrypt 0.2 refuses
134-
text with no tokens. Ordering types
132+
where that matters. Write the stored column on the left, `col @@ query`: the
133+
other order finds nothing until #1171 is fixed. Text too short to make a
134+
3-gram (empty strings, two-letter codes) cannot be stored in a `TextMatch`
135+
column yet, and a `TextMatchQuery` for it is refused too: Stack Encrypt 0.2
136+
refuses text with no tokens, on both sides. Ordering types
135137
wait for vitaminc 0.6's order encodings, float
136138
equality for its canonical float form, and `Json` for a decision on sealing a
137139
document as one leaf.

‎packages/eql/crates/eql-bindings/src/v4/text.rs‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,11 +141,15 @@ impl DomainType for TextEq {
141141
doc,
142142
doc = " See the [EQL text reference](https://cipherstash.com/docs/reference/eql/text) for SQL domain variants, operators, and query examples."
143143
)]
144-
#[cfg_attr(all(doc, feature = "stack-encrypt"), doc = "")]
144+
#[cfg_attr(doc, doc = "")]
145145
#[cfg_attr(
146146
all(doc, feature = "stack-encrypt"),
147147
doc = " Plaintext input and decrypted output: [`String`](String). With the `stack-encrypt` feature, implements `EncryptFrom<String>` and `DecryptInto<String>`."
148148
)]
149+
#[cfg_attr(
150+
all(doc, not(feature = "stack-encrypt")),
151+
doc = " Plaintext input and decrypted output: `String`. With the `stack-encrypt` feature, implements `EncryptFrom<String>` and `DecryptInto<String>`."
152+
)]
149153
#[cfg_attr(
150154
all(doc, feature = "stack-encrypt"),
151155
doc = " Encryption context: `NonEmpty<Identifier>`, constructed with `Identifier::for_column(table, column)` and stored in `i`."
@@ -480,11 +484,15 @@ impl DomainType for TextEqQuery {
480484
doc,
481485
doc = " See the [EQL text reference](https://cipherstash.com/docs/reference/eql/text) for SQL domain variants, operators, and query examples."
482486
)]
483-
#[cfg_attr(all(doc, feature = "stack-encrypt"), doc = "")]
487+
#[cfg_attr(doc, doc = "")]
484488
#[cfg_attr(
485489
all(doc, feature = "stack-encrypt"),
486490
doc = " Plaintext input: [`String`](String). With the `stack-encrypt` feature, implements `EncryptFrom<String>` to build match query terms. Query operands do not decrypt."
487491
)]
492+
#[cfg_attr(
493+
all(doc, not(feature = "stack-encrypt")),
494+
doc = " Plaintext input: `String`. With the `stack-encrypt` feature, implements `EncryptFrom<String>` to build match query terms. Query operands do not decrypt."
495+
)]
488496
#[cfg_attr(
489497
all(doc, feature = "stack-encrypt"),
490498
doc = " Encryption context: `NonEmpty<Identifier>`, constructed with `Identifier::for_column(table, column)` and stored in `i`."

‎packages/eql/crates/eql-codegen/src/bindings.rs‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -400,8 +400,6 @@ pub fn encryption_gap(
400400
.any(|t| matches!(t, Term::Ore | Term::Ope))
401401
{
402402
"ordering terms wait for vitaminc-ore, the order encodings in vitaminc 0.6 (ADR-0002)"
403-
} else if domain.terms.contains(&Term::Bloom) {
404-
"the engine derives match terms, but no EQL type is built from them yet"
405403
} else if matches!(family.name, "integer" | "bigint") && domain.terms == [Term::Hm] {
406404
"the integer equality term moves to vitaminc 0.6's shared integer-orderable/v1 PRF \
407405
domain; producing it before that release would leave stored terms unmatchable after \

0 commit comments

Comments
 (0)