Skip to content

Commit 61dac39

Browse files
authored
feat(eql): build the eql_v4 bundle beside eql_v3
ADR-0002 installs one SQL source under two names, so a column's domain says which engine wrote its terms. The build now renders the placeholder source a second time as eql_v4 and writes release/cipherstash-encrypt-v4.sql and its uninstaller. The v3 artefacts are unchanged, byte for byte. verify_no_v3_in_v4.sh fails the build if the v4 bundle names eql_v3 or checks for payload version 3: a stray v3 reference would install cleanly and then compare v4 data with v3 terms. The symbol-order check stays on v3 only, since v4 is the same files in the same order with every name renamed. v3_v4_coexistence_tests installs both bundles into one database and pins what the separate names enforce: a wrong-version payload fails the domain CHECK, and the cross-version function form is ambiguous. It also pins the gap the corrected ADR records: a plain `=` across versions falls back to jsonb = jsonb and returns no rows, silently, until cross-version blocker operators exist (#1160). Refs #1141
1 parent f4c245c commit 61dac39

4 files changed

Lines changed: 288 additions & 12 deletions

File tree

‎packages/eql/AGENTS.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,9 +38,10 @@ This project uses `mise` for task management. Common commands:
3838

3939
### Build System
4040
- Dependencies are resolved using `-- REQUIRE:` comments in SQL files
41-
- Build outputs to `release/` directory (a single installer + uninstaller, assembled from `src/v3` alone):
42-
- `cipherstash-encrypt.sql` - The sole installer: the self-contained `eql_v3` surface, globbed from `src/v3` only (no `eql_v2`; installable into a DB with no `eql_v2` present)
41+
- Build outputs to `release/` directory (one installer + uninstaller per EQL version, both rendered from `src/v3` alone):
42+
- `cipherstash-encrypt.sql` - The `eql_v3` installer: the self-contained surface, globbed from `src/v3` only (no `eql_v2`; installable into a DB with no `eql_v2` present)
4343
- `cipherstash-encrypt-uninstall.sql` - Matching uninstaller
44+
- `cipherstash-encrypt-v4.sql` / `cipherstash-encrypt-v4-uninstall.sql` - The same SQL as `eql_v4` (ADR-0002). `tasks/test/verify_no_v3_in_v4.sh` fails the build if either names `eql_v3` or checks for payload version 3. Both versions install side by side; `tests/sqlx/tests/v3_v4_coexistence_tests.rs` pins what that enforces, including the gap that a plain `=` across versions falls back to `jsonb = jsonb` and returns no rows
4445

4546
#### Schema and version placeholders
4647

@@ -53,7 +54,7 @@ The SQL source never names its schemas or its payload version literally. It writ
5354

5455
`{{prefix}}` is the schema and the start of every other name: `{{prefix}}.eq_term`, `{{prefix}}_internal.hmac_256`, `public.{{prefix}}_text_eq`. `{{eql_version}}` appears in every domain CHECK: `VALUE->>'v' = '{{eql_version}}'`. This covers every file under `src/v3/` (hand-written and generated), `src/v3/version.template`, `tasks/pin_search_path_v3.sql` and `tasks/uninstall-v3.sql`. The generator emits the placeholders too: `SCHEMA`, `INTERNAL_SCHEMA`, `PUBLIC_TYPNAME_PREFIX` and `EQL_VERSION` in `crates/eql-codegen/src/consts.rs` are placeholder strings. The Rust bindings are the exception. They describe v3 payloads and keep `eql_domains::PUBLIC_TYPNAME_PREFIX`, the literal `eql_v3_`.
5556

56-
`tasks/build.sh` renders the ordered source into `build/render/eql_v3/` (gitignored, same relative paths) with `eql-codegen render`, runs the symbol-order and installer-completeness gates on the rendered files, and assembles `release/` from them. The renderer refuses three things: an unknown placeholder, an unterminated `{{`, and a **literal `eql_v3` or `eql_v4`** anywhere in the source, comments included. A literal name renders identically for both versions, so the v4 bundle would reach into v3. Write the placeholder in prose too.
57+
`tasks/build.sh` renders the ordered source into `build/render/eql_v3/` and `build/render/eql_v4/` (gitignored, same relative paths) with `eql-codegen render`, runs the symbol-order and installer-completeness gates on the rendered files, and assembles `release/` from them. The renderer refuses three things: an unknown placeholder, an unterminated `{{`, and a **literal `eql_v3` or `eql_v4`** anywhere in the source, comments included. A literal name renders identically for both versions, so the v4 bundle would reach into v3. Write the placeholder in prose too.
5758

5859
Anything that needs the source as real SQL reads the rendered tree, not `src/`: `docs:validate:documented-sql` runs psql over `build/render/eql_v3/src`, and the Doxygen input filter (`tasks/docs/doxygen-filter.sh`) renders `eql_v3` names on the fly. A new tool that parses `src/v3` must do one or the other.
5960

‎packages/eql/tasks/build.sh‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
#!/usr/bin/env bash
22
#MISE description="Build SQL into single release file"
33
#MISE alias="b"
4-
#MISE sources=["src/v3/**/*.sql", "src/v3/version.template", "tasks/pin_search_path_v3.sql", "tasks/uninstall-v3.sql", "crates/eql-domains/src/**/*.rs", "crates/eql-codegen/src/**/*.rs", "crates/eql-codegen/templates/**", "Cargo.toml", "Cargo.lock", "crates/eql-codegen/Cargo.toml", "crates/eql-domains/Cargo.toml", "tasks/build/ordering.sh", "tasks/test/verify_symbol_order_v3.sh", "tasks/test/verify_installer_complete.sh", "tasks/test/symbol_order_allowlist.txt"]
5-
#MISE outputs=["release/cipherstash-encrypt.sql","release/cipherstash-encrypt-uninstall.sql","src/deps-ordered-v3.txt","build/render/eql_v3/deps-ordered.txt"]
4+
#MISE sources=["src/v3/**/*.sql", "src/v3/version.template", "tasks/pin_search_path_v3.sql", "tasks/uninstall-v3.sql", "crates/eql-domains/src/**/*.rs", "crates/eql-codegen/src/**/*.rs", "crates/eql-codegen/templates/**", "Cargo.toml", "Cargo.lock", "crates/eql-codegen/Cargo.toml", "crates/eql-domains/Cargo.toml", "tasks/build/ordering.sh", "tasks/test/verify_symbol_order_v3.sh", "tasks/test/verify_installer_complete.sh", "tasks/test/verify_no_v3_in_v4.sh", "tasks/test/symbol_order_allowlist.txt"]
5+
#MISE outputs=["release/cipherstash-encrypt.sql","release/cipherstash-encrypt-uninstall.sql","release/cipherstash-encrypt-v4.sql","release/cipherstash-encrypt-v4-uninstall.sql","src/deps-ordered-v3.txt","build/render/eql_v3/deps-ordered.txt","build/render/eql_v4/deps-ordered.txt"]
66
#USAGE flag "--version <version>" help="Specify release version of EQL" default="DEV"
77

88
#!/bin/bash
@@ -82,9 +82,9 @@ mv src/deps-ordered-v3.txt.tmp src/deps-ordered-v3.txt
8282

8383
# The source names its schemas and payload version by placeholder
8484
# ({{prefix}}, {{eql_version}}; ADR-0002), so it is not installable SQL until it
85-
# is rendered. Render the ordered files, the pin script and the uninstaller as
86-
# EQL v3 into build/render/eql_v3/ (same relative paths), then run every gate
87-
# and assemble the installer from the rendered tree. `eql-codegen render` fails
85+
# is rendered. Render the ordered files, the pin script and the uninstaller once
86+
# per EQL version into build/render/eql_v<N>/ (same relative paths), then run
87+
# the gates and assemble that version's installer from the rendered tree. `eql-codegen render` fails
8888
# on an unknown placeholder and on a literal eql_v3/eql_v4 in the source, and
8989
# writes nothing unless every file renders.
9090
assemble() {
@@ -98,7 +98,13 @@ assemble() {
9898
$(cat src/deps-ordered-v3.txt) tasks/pin_search_path_v3.sql tasks/uninstall-v3.sql
9999
sed "s|^|$root/|" src/deps-ordered-v3.txt > "$ordered"
100100

101-
bash tasks/test/verify_symbol_order_v3.sh "$ordered"
101+
# The symbol-order checker parses eql_v3 names. One run covers both
102+
# versions: eql_v4 is the same files in the same order, every name renamed
103+
# alike, so define-before-use holds for one exactly when it holds for the
104+
# other.
105+
if [[ "$version" == 3 ]]; then
106+
bash tasks/test/verify_symbol_order_v3.sh "$ordered"
107+
fi
102108

103109
: > "$installer"
104110
while IFS= read -r f; do
@@ -118,14 +124,22 @@ assemble() {
118124
}
119125

120126
assemble 3 release/cipherstash-encrypt.sql release/cipherstash-encrypt-uninstall.sql
127+
assemble 4 release/cipherstash-encrypt-v4.sql release/cipherstash-encrypt-v4-uninstall.sql
128+
129+
# Nothing in the v4 bundle may name v3: a stray eql_v3 reference would install
130+
# cleanly and then compare v4 data with v3 terms. The renderer already rejects
131+
# literal names in the source; this checks the artefact itself.
132+
bash tasks/test/verify_no_v3_in_v4.sh release/cipherstash-encrypt-v4.sql release/cipherstash-encrypt-v4-uninstall.sql
121133

122134
echo
123135
echo '###############################################'
124136
echo "# ✅Build succeeded"
125137
echo '###############################################'
126138
echo
127-
echo 'Installer:'
128-
echo ' release/cipherstash-encrypt.sql'
139+
echo 'Installers:'
140+
echo ' release/cipherstash-encrypt.sql (eql_v3)'
141+
echo ' release/cipherstash-encrypt-v4.sql (eql_v4)'
129142
echo
130-
echo 'Uninstaller:'
143+
echo 'Uninstallers:'
131144
echo ' release/cipherstash-encrypt-uninstall.sql'
145+
echo ' release/cipherstash-encrypt-v4-uninstall.sql'
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env bash
2+
#MISE description="Assert the eql_v4 installer and uninstaller name nothing from eql_v3 (DB-free)"
3+
#
4+
# ADR-0002 installs one SQL source under two names, so a column's domain says
5+
# which engine produced its terms. That holds only while the v4 bundle is v3's
6+
# SQL with EVERY name and version changed. A stray eql_v3 reference installs
7+
# cleanly and then quietly compares v4 data with v3 terms, or casts a v4 payload
8+
# to a v3 domain.
9+
#
10+
# `eql-codegen render` already refuses a literal eql_v3 in the source. This
11+
# checks the built artefact, so it also catches a build step that splices text
12+
# in around the renderer.
13+
#
14+
# Usage: verify_no_v3_in_v4.sh [file...] (exit 1 on a leak, 2 on bad input)
15+
# With no arguments it checks the v4 installer and uninstaller in release/.
16+
17+
set -euo pipefail
18+
19+
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
20+
cd "$REPO_ROOT"
21+
22+
if [[ $# -eq 0 ]]; then
23+
set -- release/cipherstash-encrypt-v4.sql release/cipherstash-encrypt-v4-uninstall.sql
24+
fi
25+
26+
fail=0
27+
for f in "$@"; do
28+
if [[ ! -s "$f" ]]; then
29+
echo "ERROR: $f missing or empty — run 'mise run build' first" >&2
30+
exit 2
31+
fi
32+
# Non-vacuity: a v4 bundle that never names eql_v4 was not rendered as v4.
33+
if ! grep -q 'eql_v4' "$f"; then
34+
echo "ERROR: $f never names eql_v4 — not a v4 render" >&2
35+
fail=1
36+
fi
37+
if grep -n 'eql_v3' "$f" | head -5 | sed "s|^|$f:|" | grep .; then
38+
echo "ERROR: $f names eql_v3 (first 5 shown)" >&2
39+
fail=1
40+
fi
41+
# The domain CHECKs compare the payload version. Any v3 comparison left in
42+
# the v4 bundle would admit v3 payloads into v4 columns.
43+
if grep -nE "'v'[[:space:]]*=[[:space:]]*'3'" "$f" | head -5 | sed "s|^|$f:|" | grep .; then
44+
echo "ERROR: $f checks for payload version 3 (first 5 shown)" >&2
45+
fail=1
46+
fi
47+
done
48+
49+
if [[ $fail -ne 0 ]]; then
50+
echo "v4 isolation gate FAILED" >&2
51+
exit 1
52+
fi
53+
echo "v4 isolation gate OK ($# files)"
Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
//! EQL v3 and v4 installed side by side in one database (ADR-0002).
2+
//!
3+
//! The two bundles are the same SQL under two names: `eql_v3` for terms written
4+
//! by cipherstash-client, `eql_v4` for terms written by Stack Encrypt. These
5+
//! tests pin what the separate names do and do not enforce when both are
6+
//! installed:
7+
//!
8+
//! - both install without clashing, and each owns its own domains;
9+
//! - a payload of the wrong version fails the domain CHECK on insert;
10+
//! - the function form across versions (`eql_v3.eq(v3col, v4term)`) is
11+
//! ambiguous, so it errors;
12+
//! - the operator form across versions (`v3col = v4term`) does NOT error.
13+
//! With no operator defined for the exact pair, Postgres treats both domains
14+
//! as `jsonb` and uses the built-in `jsonb = jsonb`, which returns no rows.
15+
//! That is pinned here so it cannot change unnoticed. ADR-0002 defers the
16+
//! cross-version blocker operators that would make it an error; when they
17+
//! land, those assertions flip to expecting one.
18+
//!
19+
//! The payloads are real cipherstash-client ciphertexts from the
20+
//! `eql_v3_integer` fixture. A v4 payload is the same row with its envelope
21+
//! version set to 4: the two bundles share their SQL, so only the version
22+
//! decides which domain admits it.
23+
24+
use anyhow::Result;
25+
use sqlx::PgPool;
26+
27+
const V4_INSTALLER: &str = "../../release/cipherstash-encrypt-v4.sql";
28+
29+
/// SQLSTATE `check_violation`.
30+
const CHECK_VIOLATION: &str = "23514";
31+
/// SQLSTATE `ambiguous_function`.
32+
const AMBIGUOUS_FUNCTION: &str = "42725";
33+
34+
/// Install the v4 bundle into a test database that the migrations have
35+
/// already given v3.
36+
async fn install_v4(pool: &PgPool) -> Result<()> {
37+
let sql = std::fs::read_to_string(V4_INSTALLER).unwrap_or_else(|e| {
38+
panic!(
39+
"failed to read {V4_INSTALLER}: {e} — run `mise run build` \
40+
(or, in CI, ensure the nextest-archive artifact shipped release/*.sql)"
41+
)
42+
});
43+
sqlx::raw_sql(&sql).execute(pool).await?;
44+
Ok(())
45+
}
46+
47+
/// One `integer_eq` column per version, each holding the fixture row for 42,
48+
/// written as that version's payload.
49+
async fn tables_with_one_row(pool: &PgPool) -> Result<()> {
50+
sqlx::raw_sql(
51+
"CREATE TABLE t3 (c public.eql_v3_integer_eq);
52+
CREATE TABLE t4 (c public.eql_v4_integer_eq);
53+
INSERT INTO t3 SELECT payload::public.eql_v3_integer_eq
54+
FROM fixtures.eql_v3_integer WHERE plaintext = 42;
55+
INSERT INTO t4 SELECT jsonb_set(payload, '{v}', '4')::public.eql_v4_integer_eq
56+
FROM fixtures.eql_v3_integer WHERE plaintext = 42;",
57+
)
58+
.execute(pool)
59+
.await?;
60+
Ok(())
61+
}
62+
63+
/// The term-only query operand for 42 (no ciphertext `c`), as `version`.
64+
fn query_term(version: u8) -> String {
65+
format!(
66+
"(SELECT jsonb_set(payload - 'c', '{{v}}', '{version}') \
67+
FROM fixtures.eql_v3_integer WHERE plaintext = 42)::eql_v{version}.query_integer_eq"
68+
)
69+
}
70+
71+
fn sqlstate(err: &sqlx::Error) -> Option<String> {
72+
match err {
73+
sqlx::Error::Database(e) => e.code().map(|c| c.into_owned()),
74+
_ => None,
75+
}
76+
}
77+
78+
#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer")))]
79+
async fn v3_and_v4_install_side_by_side(pool: PgPool) -> Result<()> {
80+
install_v4(&pool).await?;
81+
82+
let schemas: Vec<String> = sqlx::query_scalar(
83+
"SELECT nspname::text FROM pg_namespace WHERE nspname LIKE 'eql\\_v_%' ORDER BY 1",
84+
)
85+
.fetch_all(&pool)
86+
.await?;
87+
assert_eq!(
88+
schemas,
89+
["eql_v3", "eql_v3_internal", "eql_v4", "eql_v4_internal"]
90+
);
91+
92+
let (v3, v4): (i64, i64) = sqlx::query_as(
93+
"SELECT count(*) FILTER (WHERE typname LIKE 'eql\\_v3\\_%'),
94+
count(*) FILTER (WHERE typname LIKE 'eql\\_v4\\_%')
95+
FROM pg_type WHERE typnamespace = 'public'::regnamespace AND typtype = 'd'",
96+
)
97+
.fetch_one(&pool)
98+
.await?;
99+
assert!(v3 > 0, "v3 installed no public domains");
100+
assert_eq!(v3, v4, "each version owns the same set of public domains");
101+
102+
let (a, b): (String, String) = sqlx::query_as("SELECT eql_v3.version(), eql_v4.version()")
103+
.fetch_one(&pool)
104+
.await?;
105+
assert_eq!(a, b, "both bundles come from one build");
106+
Ok(())
107+
}
108+
109+
#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer")))]
110+
async fn each_version_matches_its_own_terms(pool: PgPool) -> Result<()> {
111+
install_v4(&pool).await?;
112+
tables_with_one_row(&pool).await?;
113+
114+
for (table, version) in [("t3", 3), ("t4", 4)] {
115+
let n: i64 = sqlx::query_scalar(&format!(
116+
"SELECT count(*) FROM {table} WHERE c = {}",
117+
query_term(version)
118+
))
119+
.fetch_one(&pool)
120+
.await?;
121+
assert_eq!(n, 1, "{table} = its own version's term must match");
122+
}
123+
Ok(())
124+
}
125+
126+
#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer")))]
127+
async fn wrong_version_insert_fails_the_domain_check(pool: PgPool) -> Result<()> {
128+
install_v4(&pool).await?;
129+
130+
for (domain, version) in [
131+
("public.eql_v3_integer_eq", 4),
132+
("public.eql_v4_integer_eq", 3),
133+
] {
134+
let err = sqlx::query(&format!(
135+
"SELECT jsonb_set(payload, '{{v}}', '{version}')::{domain} \
136+
FROM fixtures.eql_v3_integer WHERE plaintext = 42"
137+
))
138+
.execute(&pool)
139+
.await
140+
.expect_err("a payload of the other version must not fit the domain");
141+
assert_eq!(
142+
sqlstate(&err).as_deref(),
143+
Some(CHECK_VIOLATION),
144+
"a version-{version} payload cast to {domain}: {err}"
145+
);
146+
}
147+
Ok(())
148+
}
149+
150+
#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer")))]
151+
async fn function_form_across_versions_is_ambiguous(pool: PgPool) -> Result<()> {
152+
install_v4(&pool).await?;
153+
tables_with_one_row(&pool).await?;
154+
155+
for (table, function, term) in [("t3", "eql_v3.eq", 4), ("t4", "eql_v4.eq", 3)] {
156+
let err = sqlx::query(&format!(
157+
"SELECT count(*) FROM {table} WHERE {function}(c, {})",
158+
query_term(term)
159+
))
160+
.execute(&pool)
161+
.await
162+
.expect_err("a cross-version function call must not resolve");
163+
assert_eq!(
164+
sqlstate(&err).as_deref(),
165+
Some(AMBIGUOUS_FUNCTION),
166+
"{function}({table}.c, v{term} term): {err}"
167+
);
168+
}
169+
Ok(())
170+
}
171+
172+
/// Pins a known gap: see the module docs. When cross-version blocker operators
173+
/// exist, these comparisons raise instead, and this test changes with them.
174+
#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer")))]
175+
async fn operator_across_versions_returns_no_rows(pool: PgPool) -> Result<()> {
176+
install_v4(&pool).await?;
177+
tables_with_one_row(&pool).await?;
178+
179+
for (table, term) in [("t3", 4), ("t4", 3)] {
180+
let n: i64 = sqlx::query_scalar(&format!(
181+
"SELECT count(*) FROM {table} WHERE c = {}",
182+
query_term(term)
183+
))
184+
.fetch_one(&pool)
185+
.await?;
186+
assert_eq!(n, 0, "{table}.c = v{term} term runs, and finds nothing");
187+
188+
// Why: no EQL operator takes this pair, so the planner chose the
189+
// built-in jsonb equality on the two domains' base type.
190+
let plan: Vec<String> = sqlx::query_scalar(&format!(
191+
"EXPLAIN (COSTS OFF) SELECT * FROM {table} WHERE c = {}",
192+
query_term(term)
193+
))
194+
.fetch_all(&pool)
195+
.await?;
196+
let plan = plan.join("\n");
197+
assert!(
198+
plan.contains("(c)::jsonb = "),
199+
"{table}.c = v{term} term should fall back to jsonb = jsonb:\n{plan}"
200+
);
201+
}
202+
203+
let joined: i64 = sqlx::query_scalar("SELECT count(*) FROM t3 JOIN t4 ON t3.c = t4.c")
204+
.fetch_one(&pool)
205+
.await?;
206+
assert_eq!(joined, 0, "a v3 column joined to a v4 column finds nothing");
207+
Ok(())
208+
}

0 commit comments

Comments
 (0)