Skip to content

Commit c5cf0c9

Browse files
authored
Merge pull request #1165 from cipherstash/test/golang-eql-v4
test(golang): prove the EQL v4 types from Go, against Rust and PostgreSQL
2 parents 0dc292e + c242932 commit c5cf0c9

11 files changed

Lines changed: 1191 additions & 19 deletions

File tree

‎.github/workflows/tests-golang.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ on:
3636
# The eql guest build links eql-bindings.
3737
- packages/eql/crates/eql-bindings/**
3838
- packages/eql/crates/eql-domains/**
39+
# The Go tests read the cross-language fixtures the EQL suite writes,
40+
# which are derived under EQL's lockfile.
41+
- packages/eql/tests/encryption/**
42+
- packages/eql/Cargo.lock
3943
- languages/golang/**
4044
- scripts/check-wasm-imports.py
4145
- scripts/go-binding-test.sh
@@ -61,6 +65,10 @@ on:
6165
# The eql guest build links eql-bindings.
6266
- packages/eql/crates/eql-bindings/**
6367
- packages/eql/crates/eql-domains/**
68+
# The Go tests read the cross-language fixtures the EQL suite writes,
69+
# which are derived under EQL's lockfile.
70+
- packages/eql/tests/encryption/**
71+
- packages/eql/Cargo.lock
6472
- languages/golang/**
6573
- scripts/check-wasm-imports.py
6674
- scripts/go-binding-test.sh
@@ -133,6 +141,7 @@ jobs:
133141
.
134142
languages/golang/encrypt/guest
135143
languages/golang/auth/guest
144+
packages/eql
136145
137146
# The HTTP-free core compiles for wasm32-wasip1 with no JS-host backend
138147
# and no native HTTP/TLS stack in its graph: the invariant the wazero
@@ -220,6 +229,18 @@ jobs:
220229
echo "RLIMIT_MEMLOCK: $(ulimit -l) KiB"
221230
mise run go:test
222231
232+
# The Rust suite opens the committed v4_rows.json, whose ciphertexts the
233+
# Go SDK sealed when the file was last regenerated. This also hands it
234+
# the rows the Go SDK seals in this run, so a Go guest and a native
235+
# Rust build that stop agreeing on today's ciphertexts fail here rather
236+
# than when someone next regenerates the file. The toolchain is the
237+
# root's pin, as for the guests.
238+
- name: Rows the Go SDK seals today open in Rust
239+
run: |
240+
rows="$RUNNER_TEMP/v4_rows.json"
241+
(cd languages/golang && CGO_ENABLED=0 EQL_V4_ROWS_OUT="$rows" go test -count=1 -run TestV4RowsMatchTheCommittedFixture ./encrypt)
242+
EQL_V4_ROWS="$rows" cargo test --locked --manifest-path packages/eql/Cargo.toml -p eql-encryption-tests --test v4_types rows_the_go_sdk_sealed_open_in_rust
243+
223244
# What is encrypted is fixed before the program ships: every generated
224245
# file is committed, and this fails when `go generate` would change
225246
# one. It runs the real stashgen against the guest just built, so it

‎biome.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"!languages/typescript/packages/protect-ffi/src/eql-v3-types",
1313
"!packages/stack-encrypt/tests/fixtures/record_lowering.json",
1414
"!packages/eql/tests/encryption/fixtures/ste_vec_selectors.json",
15+
"!packages/eql/tests/encryption/fixtures/v4_rows.json",
1516
"!packages/eql/crates/eql-bindings/bindings",
1617
"!packages/eql/crates/eql-bindings/schema",
1718
"!packages/eql/packages/eql/src/generated",

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ Your program calls those functions, and it never builds or names a plan.
4444
```
4545

4646
6. Store the encrypted type.
47-
Each `encrypt_into` field is one EQL column: `eql.TextEq` implements `driver.Valuer` and `sql.Scanner`, so a database library binds and scans it as the JSON a `public.eql_v4_text_eq` column holds. Stack Encrypt writes EQL v4; see the `encrypt` package README.
47+
Each `encrypt_into` field is one EQL column: `eql.TextEq` implements `driver.Valuer` and `sql.Scanner`, so a database library binds and scans it as the JSON a `public.eql_v4_text_eq` column holds. Stack Encrypt writes EQL v4; see the `encrypt` package README for the EQL types it produces and the Go field type each takes.
4848

4949
7. Run the generator again after each change to the struct or to a tag.
5050
A change to the fields of the struct stops the build until you do.

‎languages/golang/encrypt/README.md‎

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,15 @@ The package reference is on [pkg.go.dev]; the generator's reference is
3131
type User struct {
3232
_ struct{} `stash:"context=users"`
3333
ID int64 `stash:"id,passthrough"`
34-
Email string `stash:"email,encrypt,index=equality;match"`
35-
Age uint32 `stash:"age,encrypt,index=equality;ore"`
34+
Email string `stash:"email,encrypt_into=TextEq"`
35+
Name string `stash:"name,encrypt_into=TextMatch"`
36+
Age int32 `stash:"age,encrypt_into=Integer"`
3637
}
3738
```
3839

40+
`encrypt_into=` names the EQL type the field is stored as. The Go field
41+
type must match it; see [EQL columns](#eql-columns).
42+
3943
3. Run the generator. It writes `user_stash.go` beside the struct.
4044

4145
```sh
@@ -56,17 +60,25 @@ The package reference is on [pkg.go.dev]; the generator's reference is
5660

5761
encrypted, err := users.Encrypt(ctx, cipher, people)
5862
opened, err := users.Decrypt(ctx, cipher, encrypted)
59-
term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com")
63+
query, err := users.Fields.Email.Query(ctx, cipher, "bob@example.com")
6064
```
6165

6266
`Encrypt` and `Decrypt` send one ZeroKMS request for each 500 sealed
6367
values in the batch, plus one request the first time a keyset is used.
6468

65-
6. Store the encrypted type. Each sealed field is one or more byte columns:
66-
`e.Email.Ciphertext`, `e.Email.Equality`, `e.Email.Match`. `Ciphertext`
67-
and each term type implement `driver.Valuer` and `sql.Scanner`, so a
68-
database library binds and scans each column as bytes; map each one to
69-
its own column.
69+
6. Store the encrypted type. Each `encrypt_into` field is one EQL v4 column:
70+
`eql.TextEq` implements `driver.Valuer` and `sql.Scanner`, so a database
71+
library binds and scans it as the JSON a `public.eql_v4_text_eq` column
72+
holds. Install EQL v4 into PostgreSQL first, with the
73+
`eql-v4-install.sql` attached to the release. A query casts its parameter
74+
to the column's query type:
75+
76+
```sql
77+
SELECT id, email, name, age FROM users WHERE email = $1::eql_v4.query_text_eq
78+
```
79+
80+
A field tagged `encrypt` or `index=` stores its ciphertext and each term
81+
as separate byte columns instead; see [What is stored](#what-is-stored).
7082

7183
7. Run the generator again after each change to the struct or to a tag. A
7284
change to the fields of the struct stops the build until you do.
@@ -80,8 +92,8 @@ The package reference is on [pkg.go.dev]; the generator's reference is
8092
`git diff` sees only files git already tracks; the `git status` check
8193
also fails on a generated file that was never committed.
8294

83-
The rest of this file is the reference. [`example/`](example/) is the eight
84-
steps as a program.
95+
The rest of this file is the reference. [`example/`](example/) is a program
96+
that stores separate byte columns (`index=`) against real ZeroKMS.
8597

8698
## Connect
8799

@@ -282,7 +294,27 @@ of EQL's SQL; the `eql_v3` columns the TypeScript SDK writes hold
282294
cipherstash-client's terms, which these never match. Install EQL v4 into
283295
PostgreSQL first, with the `eql-v4-install.sql` attached to each Go release
284296
(`psql -f eql-v4-install.sql`); `stash eql install` installs EQL v3 only, and
285-
an `eql_v4_*` column needs the v4 domains. The guest builds the value: the generated code
297+
an `eql_v4_*` column needs the v4 domains.
298+
299+
The EQL types the engine produces, and the Go field types each takes:
300+
301+
| EQL type | Go field | Queries |
302+
|---|---|---|
303+
| `TextEq` | `string` | `=`, `<>` |
304+
| `TextMatch` | `string` | `@@` (substring match, 3-grams, case-insensitive) |
305+
| `Text` | `string` | none: stored and read only |
306+
| `Integer` | `int32` | none |
307+
| `Bigint` | `int64` | none |
308+
| `Boolean` | `bool` | none |
309+
| `Real` | `float32` | none |
310+
| `Double` | `float64` | none |
311+
312+
A `TextMatch` value must be at least three characters long: shorter text has
313+
no 3-grams to index and is refused. `IntegerEq` and `BigintEq` are held back
314+
until the engine's integer equality term settles in vitaminc 0.6. The rest of
315+
the catalog (ordering, dates, decimals, JSON) is listed in
316+
`eql.Types` with the reason it is not produced yet, and `stashgen` refuses it
317+
with that reason. The guest builds the value: the generated code
286318
imports `encrypt/eql`, which embeds the build of the engine that holds the
287319
EQL types and registers it on import, so the program runs that build and
288320
stores what it returns. `Fields.Email.Query` returns the `eql.TextEqQuery`

‎languages/golang/encrypt/doc.go‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,15 @@
1515
// type User struct {
1616
// _ struct{} `stash:"context=users"`
1717
// ID int64 `stash:"id,passthrough"`
18-
// Email string `stash:"email,encrypt,index=equality;match"`
19-
// Age uint32 `stash:"age,encrypt,index=equality;ore"`
18+
// Email string `stash:"email,encrypt_into=TextEq"`
19+
// Name string `stash:"name,encrypt_into=TextMatch"`
20+
// Age int32 `stash:"age,encrypt_into=Integer"`
2021
// }
2122
//
23+
// encrypt_into names the EQL type the field is stored as. The Go field type
24+
// must match it: TextEq, TextMatch and Text take a string, Integer an int32,
25+
// Bigint an int64, Boolean a bool, Real a float32 and Double a float64.
26+
//
2227
// 3. Run the generator. It writes user_stash.go beside the struct:
2328
//
2429
// go generate ./...
@@ -35,15 +40,22 @@
3540
// cipher := client.Keyset(encrypt.KeysetName("tenant-42"))
3641
// encrypted, err := users.Encrypt(ctx, cipher, people) // []users.EncryptedUser
3742
// opened, err := users.Decrypt(ctx, cipher, encrypted) // []users.User
38-
// term, err := users.Fields.Email.Equality(ctx, cipher, "bob@example.com")
43+
// query, err := users.Fields.Email.Query(ctx, cipher, "bob@example.com")
3944
//
4045
// Encrypt and Decrypt send one ZeroKMS request for each 500 sealed values in
4146
// the batch, plus one the first time a keyset is used.
4247
//
43-
// 6. Store the encrypted type. Each sealed field is one or more columns of
44-
// bytes ([Ciphertext] and the term types), each implementing driver.Valuer
45-
// and sql.Scanner, so a database library binds and scans each one as bytes;
46-
// map each one to its own column.
48+
// 6. Store the encrypted type. Each encrypt_into field is one EQL v4 column
49+
// (an eql.TextEq goes in a public.eql_v4_text_eq column), implementing
50+
// driver.Valuer and sql.Scanner, so a database library binds and scans it
51+
// like any other value. Install EQL v4 into PostgreSQL first, with the
52+
// eql-v4-install.sql attached to the release. A query casts its parameter to
53+
// the column's query type:
54+
//
55+
// SELECT id, email, name, age FROM users WHERE email = $1::eql_v4.query_text_eq
56+
//
57+
// A field tagged encrypt or index= stores its ciphertext ([Ciphertext]) and
58+
// each term as separate byte columns instead; see Terms below.
4759
//
4860
// 7. Run the generator again after each change to the struct or to a tag. A
4961
// change to the fields of the struct stops the build until you do.

0 commit comments

Comments
 (0)