Skip to content

Commit 0dc292e

Browse files
authored
Merge pull request #1164 from cipherstash/feat/eql-v4-text-match
feat(eql-bindings): produce EQL v4 TextMatch from Stack Encrypt match terms
2 parents 2d899c0 + c4ec6dd commit 0dc292e

20 files changed

Lines changed: 381 additions & 46 deletions

File tree

‎.changeset/eql-v4-bindings.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
'@cipherstash/eql': minor
33
---
44

5-
**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.
5+
**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.
66

77
Nothing a released version shipped has changed. The v3 types, the v3 SQL and the TypeScript package are as they were; the `stack-encrypt` derives on the v3 text types and the `v3::targets` module were added after the last release and are moved to v4 here, not removed from a release.

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,14 +153,16 @@ The engine produces these EQL types today, each from one Go field type:
153153

154154
| `encrypt_into` | Go field |
155155
|---|---|
156-
| `TextEq`, `Text` | `string` |
156+
| `TextEq`, `Text`, `TextMatch` | `string` |
157157
| `Integer` | `int32` |
158158
| `Bigint` | `int64` |
159159
| `Boolean` | `bool` |
160160
| `Real` | `float32` |
161161
| `Double` | `float64` |
162162

163163
The width must match: an `int` or `int64` field is not an `Integer`, and a `float64` field is not a `Real`.
164+
A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too.
165+
Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed.
164166
`encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet.
165167
Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`.
166168

‎languages/golang/cmd/stashgen/main_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -142,15 +142,15 @@ func TestRunAsksTheEmbeddedEngine(t *testing.T) {
142142
t.Fatalf("the generated file does not name the EQL type:\n%s", written)
143143
}
144144
// A type the engine cannot produce yet: refused by name, nothing written.
145-
unproducible := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt_into=TextMatch", 1)
145+
unproducible := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt_into=TextOrdOre", 1)
146146
dir = writeModule(t, map[string]string{"model.go": unproducible})
147147
stderr.Reset()
148148
if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 {
149149
t.Fatalf("exit %d, want 1\n%s", code, stderr.String())
150150
}
151151
// Refused by the generator with the engine's reason, before the engine
152152
// is asked: se_targets lists the type as not producible.
153-
if !strings.Contains(stderr.String(), "cannot produce the EQL type TextMatch yet") {
153+
if !strings.Contains(stderr.String(), "cannot produce the EQL type TextOrdOre yet") {
154154
t.Fatalf("stderr = %q", stderr.String())
155155
}
156156
if _, err := os.Stat(filepath.Join(dir, "user_stash.go")); !os.IsNotExist(err) {

‎languages/golang/encrypt/eql/eql_gen.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎languages/golang/encrypt/eql/eql_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) {
1919
t.Errorf("%s: an unproducible type has a reason", typ.Name)
2020
}
2121
}
22-
want := []string{"Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"}
22+
want := []string{"Integer", "Bigint", "Text", "TextEq", "TextMatch", "Boolean", "Real", "Double"}
2323
if len(producible) != len(want) {
2424
t.Fatalf("producible = %v, want %v", producible, want)
2525
}

‎languages/golang/encrypt/guest/src/targets.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,16 @@ mod tests {
181181
.collect();
182182
assert_eq!(
183183
producible,
184-
["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"],
184+
[
185+
"Integer",
186+
"Bigint",
187+
"Text",
188+
"TextEq",
189+
"TextMatch",
190+
"Boolean",
191+
"Real",
192+
"Double"
193+
],
185194
"the engine's producible types, in catalog order"
186195
);
187196
assert!(matches!(

‎languages/golang/encrypt/guest/tests/native_ops.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1368,7 +1368,16 @@ fn targets_lists_what_the_build_holds() {
13681368
.collect();
13691369
assert_eq!(
13701370
producible,
1371-
["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"],
1371+
[
1372+
"Integer",
1373+
"Bigint",
1374+
"Text",
1375+
"TextEq",
1376+
"TextMatch",
1377+
"Boolean",
1378+
"Real",
1379+
"Double"
1380+
],
13721381
"the engine's producible types, in catalog order"
13731382
);
13741383
}

‎languages/golang/stashgen/enginetest/enginetest.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ var eqlTypes = []struct {
3232
{stashgen.EQLType{Name: "Bigint", Plaintext: stashgen.KindInt, Producible: true}, "int64"},
3333
{stashgen.EQLType{Name: "Text", Plaintext: stashgen.KindString, Producible: true}, ""},
3434
{stashgen.EQLType{Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true}, ""},
35+
{stashgen.EQLType{Name: "TextMatch", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexMatch}, Query: "TextMatchQuery", Producible: true}, ""},
3536
{stashgen.EQLType{Name: "Boolean", Plaintext: stashgen.KindBool, Producible: true}, ""},
3637
{stashgen.EQLType{Name: "Real", Plaintext: stashgen.KindFloat, Producible: true}, "float32"},
3738
{stashgen.EQLType{Name: "Double", Plaintext: stashgen.KindFloat, Producible: true}, "float64"},

‎languages/golang/stashgen/refusal_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ func TestRefusals(t *testing.T) {
9191
{"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"},
9292
{"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"},
9393
{"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"},
94-
{"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "has no EQL type TextMatch"},
94+
{"an EQL type the engine cannot produce yet", user(ctx + "\tAge int32 `stash:\"age,encrypt_into=IntegerEq\"`"), stashgen.Request{Type: "User"}, "Age", "has no EQL type IntegerEq"},
9595
{"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"},
9696
{"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"},
9797
{"an index option a query term cannot use", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "a query term uses only the default index options"},

‎languages/golang/stashgen/testdata/cases/users/model.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,6 @@ type User struct {
88
_ struct{} `stash:"context=users"`
99
ID int64 `stash:"id,passthrough" db:"id" gorm:"primaryKey"`
1010
Email string `stash:"email,encrypt_into=TextEq" db:"email"`
11-
Name string `stash:"name,encrypt_into=TextEq" db:"name"`
11+
Name string `stash:"name,encrypt_into=TextMatch" db:"name"`
1212
Internal string `stash:"-"`
1313
}

0 commit comments

Comments
 (0)