Skip to content

Commit 0b15a90

Browse files
authored
Merge pull request #1545 from cipherstash/claude/project-thread-62v10t
fix(eql): align JSON array flag with SQL and handle empty arrays
2 parents 241fc05 + ce65ffb commit 0b15a90

15 files changed

Lines changed: 233 additions & 51 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@cipherstash/eql': patch
3+
---
4+
5+
Encrypted JSON arrays now agree between the Rust/TypeScript bindings and the SQL. `SteVecDocument` models the root `a` array flag that the SQL root validator accepts and the array accessors require, so an array document the database accepts also parses in the bindings. The `a` flag on documents and entries is now optional but not nullable in the generated TypeScript and JSON Schema (`a?: boolean`), matching the SQL CHECK; a `null` is rejected instead of failing later with a cast error. `eql_v3.jsonb_array_length` returns 0 for an empty array (it returned NULL), and `eql_v3.jsonb_array_elements` and `eql_v3.jsonb_document_contains(jsonb[], jsonb)` no longer raise "upper bound of FOR loop cannot be null" on one.

‎packages/eql/crates/eql-bindings/bindings/v3/SteVecDocument.ts‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,20 @@ import type { SteVecForm } from "./SteVecForm";
77

88
/**
99
* `public.eql_v3_json_search` — a SteVec encrypted-JSON document
10-
* (`{v, k, i, h, sv:[entry]}`, no root ciphertext — the root document
10+
* (`{v, k, i, h, sv:[entry]}` plus an optional array-root flag `a`, no root ciphertext — the root document
1111
* ciphertext lives on the root sv entry). Strict. `k` is the `"sv"` form
1212
* discriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],
1313
* stored once for the whole document (every entry encrypts under the
1414
* document's single data key, with per-entry nonces derived from the
1515
* entries' selectors).
1616
*/
17-
export type SteVecDocument = { v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>, };
17+
export type SteVecDocument = { v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>,
18+
/**
19+
* Array-root marker. The SQL root validator allows it and the array
20+
* accessors (`jsonb_array_length`, `jsonb_array_elements`, `->(doc, int)`)
21+
* REQUIRE it (`eql_v3_internal.is_ste_vec_array`), so a document the
22+
* database accepts must also parse here. Optional and non-nullable: SQL
23+
* rejects `"a": null` (`jsonb_typeof(...) = 'boolean'`), so the schema
24+
* and deserializer reject it too.
25+
*/
26+
a?: boolean, };

‎packages/eql/crates/eql-bindings/bindings/v3/SteVecEntry.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,11 @@ import type { Selector } from "./Selector";
1111
* selector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the
1212
* decryption unit is `h` + `s` + `c`, with the document `h` grafted onto
1313
* extracted entries by `->`), an optional array-membership marker `a`
14-
* (emitted only when true), and — for ordered (number/string) path entries
14+
* (a boolean; producers emit it only when true), and — for ordered (number/string) path entries
1515
* only — the `op` ordering term. Value entries (value-inclusive selectors)
1616
* and non-orderable path entries are term-less: exact matching is selector
1717
* presence, so no per-entry equality term exists (`hm` is retired). The
1818
* optional `i`/`v`/`h` fields model metadata grafted onto an entry returned by
1919
* `->`; naming them explicitly keeps unknown keys rejectable.
2020
*/
21-
export type SteVecEntry = { s: Selector, c: EntryCiphertext, a?: boolean | null, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, };
21+
export type SteVecEntry = { s: Selector, c: EntryCiphertext, a?: boolean, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, };

‎packages/eql/crates/eql-bindings/schema/v3/eql_v3_json_entry.json‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,10 @@
4444
"$id": "https://schemas.cipherstash.com/eql/v3/eql_v3_json_entry.json",
4545
"$schema": "https://json-schema.org/draft/2020-12/schema",
4646
"additionalProperties": false,
47-
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(emitted only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
47+
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(a boolean; producers emit it only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
4848
"properties": {
4949
"a": {
50-
"type": [
51-
"boolean",
52-
"null"
53-
]
50+
"type": "boolean"
5451
},
5552
"c": {
5653
"$ref": "#/$defs/EntryCiphertext"

‎packages/eql/crates/eql-bindings/schema/v3/eql_v3_json_search.json‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -42,13 +42,10 @@
4242
},
4343
"SteVecEntry": {
4444
"additionalProperties": false,
45-
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(emitted only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
45+
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(a boolean; producers emit it only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
4646
"properties": {
4747
"a": {
48-
"type": [
49-
"boolean",
50-
"null"
51-
]
48+
"type": "boolean"
5249
},
5350
"c": {
5451
"$ref": "#/$defs/EntryCiphertext"
@@ -112,8 +109,12 @@
112109
"$id": "https://schemas.cipherstash.com/eql/v3/eql_v3_json_search.json",
113110
"$schema": "https://json-schema.org/draft/2020-12/schema",
114111
"additionalProperties": false,
115-
"description": "`public.eql_v3_json_search` — a SteVec encrypted-JSON document\n(`{v, k, i, h, sv:[entry]}`, no root ciphertext — the root document\nciphertext lives on the root sv entry). Strict. `k` is the `\"sv\"` form\ndiscriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],\nstored once for the whole document (every entry encrypts under the\ndocument's single data key, with per-entry nonces derived from the\nentries' selectors).",
112+
"description": "`public.eql_v3_json_search` — a SteVec encrypted-JSON document\n(`{v, k, i, h, sv:[entry]}` plus an optional array-root flag `a`, no root ciphertext — the root document\nciphertext lives on the root sv entry). Strict. `k` is the `\"sv\"` form\ndiscriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],\nstored once for the whole document (every entry encrypts under the\ndocument's single data key, with per-entry nonces derived from the\nentries' selectors).",
116113
"properties": {
114+
"a": {
115+
"description": "Array-root marker. The SQL root validator allows it and the array\naccessors (`jsonb_array_length`, `jsonb_array_elements`, `->(doc, int)`)\nREQUIRE it (`eql_v3_internal.is_ste_vec_array`), so a document the\ndatabase accepts must also parse here. Optional and non-nullable: SQL\nrejects `\"a\": null` (`jsonb_typeof(...) = 'boolean'`), so the schema\nand deserializer reject it too.",
116+
"type": "boolean"
117+
},
117118
"h": {
118119
"$ref": "#/$defs/KeyHeader"
119120
},

‎packages/eql/crates/eql-bindings/src/v3/json.rs‎

Lines changed: 39 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ impl JsonSchema for SteVecForm {
8080
}
8181

8282
/// `public.eql_v3_json_search` — a SteVec encrypted-JSON document
83-
/// (`{v, k, i, h, sv:[entry]}`, no root ciphertext — the root document
83+
/// (`{v, k, i, h, sv:[entry]}` plus an optional array-root flag `a`, no root ciphertext — the root document
8484
/// ciphertext lives on the root sv entry). Strict. `k` is the `"sv"` form
8585
/// discriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],
8686
/// stored once for the whole document (every entry encrypts under the
@@ -100,13 +100,37 @@ pub struct SteVecDocument {
100100
pub i: Identifier,
101101
pub h: KeyHeader,
102102
pub sv: Vec<SteVecEntry>,
103+
/// Array-root marker. The SQL root validator allows it and the array
104+
/// accessors (`jsonb_array_length`, `jsonb_array_elements`, `->(doc, int)`)
105+
/// REQUIRE it (`eql_v3_internal.is_ste_vec_array`), so a document the
106+
/// database accepts must also parse here. Optional and non-nullable: SQL
107+
/// rejects `"a": null` (`jsonb_typeof(...) = 'boolean'`), so the schema
108+
/// and deserializer reject it too.
109+
#[serde(
110+
default,
111+
skip_serializing_if = "Option::is_none",
112+
deserialize_with = "deserialize_present_bool"
113+
)]
114+
#[schemars(with = "bool")]
115+
#[ts(optional, type = "boolean")]
116+
pub a: Option<bool>,
117+
}
118+
119+
/// Deserialize an optional boolean that, when present, must be a real boolean.
120+
/// `Option<bool>`'s default deserializer maps JSON `null` to `None`; the SQL
121+
/// CHECKs reject `null` for `a`, so the wire types must too.
122+
fn deserialize_present_bool<'de, D>(deserializer: D) -> Result<Option<bool>, D::Error>
123+
where
124+
D: serde::Deserializer<'de>,
125+
{
126+
bool::deserialize(deserializer).map(Some)
103127
}
104128

105129
/// `public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a
106130
/// selector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the
107131
/// decryption unit is `h` + `s` + `c`, with the document `h` grafted onto
108132
/// extracted entries by `->`), an optional array-membership marker `a`
109-
/// (emitted only when true), and — for ordered (number/string) path entries
133+
/// (a boolean; producers emit it only when true), and — for ordered (number/string) path entries
110134
/// only — the `op` ordering term. Value entries (value-inclusive selectors)
111135
/// and non-orderable path entries are term-less: exact matching is selector
112136
/// presence, so no per-entry equality term exists (`hm` is retired). The
@@ -123,13 +147,20 @@ pub struct SteVecDocument {
123147
pub struct SteVecEntry {
124148
pub s: Selector,
125149
pub c: EntryCiphertext,
126-
// `#[ts(optional = nullable)]` emits `a?: boolean | null` — ts-rs does NOT
150+
// `#[ts(optional, type = "boolean")]` emits `a?: boolean` — ts-rs does NOT
127151
// infer optionality from serde `default`/`skip_serializing_if` (10.1: only an
128-
// explicit `#[ts(optional)]`/`optional = nullable` does), so without this the
129-
// TS binding would render `a` REQUIRED and drift from the JSON Schema, which
130-
// excludes `a` from `required`. `= nullable` keeps the `| null`.
131-
#[serde(default, skip_serializing_if = "Option::is_none")]
132-
#[ts(optional = nullable)]
152+
// explicit `#[ts(optional)]` does), so without this the TS binding would
153+
// render `a` REQUIRED and drift from the JSON Schema, which excludes `a`
154+
// from `required`. Non-nullable: the SQL entry CHECK requires
155+
// `jsonb_typeof(VALUE->'a') = 'boolean'`, so `"a": null` is rejected there
156+
// and must not be advertised (or accepted) here.
157+
#[serde(
158+
default,
159+
skip_serializing_if = "Option::is_none",
160+
deserialize_with = "deserialize_present_bool"
161+
)]
162+
#[schemars(with = "bool")]
163+
#[ts(optional, type = "boolean")]
133164
pub a: Option<bool>,
134165
#[serde(default, skip_serializing_if = "Option::is_none")]
135166
#[ts(optional)]

‎packages/eql/crates/eql-bindings/tests/v3_conformance.rs‎

Lines changed: 63 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,41 @@ fn stevec_document_round_trips_and_enforces_envelope() {
459459
assert!(serde_json::from_value::<SteVecDocument>(extra).is_err());
460460
}
461461

462+
#[test]
463+
fn stevec_array_root_flag_matches_sql_contract() {
464+
use eql_bindings::v3::json::{SteVecDocument, SteVecEntry};
465+
// The SQL root validator allows root `a`, and the array accessors require
466+
// it, so an array document the database accepts must parse and round-trip
467+
// here — including an empty one.
468+
let wire = json!({
469+
"v": 3, "k": "sv", "i": { "t": "users", "c": "tags" },
470+
"h": "kh", "a": true, "sv": []
471+
});
472+
let parsed: SteVecDocument = serde_json::from_value(wire.clone()).unwrap();
473+
assert_eq!(parsed.a, Some(true));
474+
assert_eq!(serde_json::to_value(&parsed).unwrap(), wire);
475+
// Absent `a` stays absent on re-serialization.
476+
let mut no_a = wire.clone();
477+
no_a.as_object_mut().unwrap().remove("a");
478+
let parsed: SteVecDocument = serde_json::from_value(no_a.clone()).unwrap();
479+
assert_eq!(parsed.a, None);
480+
assert_eq!(serde_json::to_value(&parsed).unwrap(), no_a);
481+
// SQL rejects a non-boolean `a` (null included) at root and entry level.
482+
for bad in [json!(null), json!("true"), json!(1)] {
483+
let mut w = wire.clone();
484+
w["a"] = bad.clone();
485+
assert!(
486+
serde_json::from_value::<SteVecDocument>(w).is_err(),
487+
"root a = {bad} must fail"
488+
);
489+
assert!(
490+
serde_json::from_value::<SteVecEntry>(json!({ "s": "sel", "c": "ct", "a": bad }))
491+
.is_err(),
492+
"entry a = {bad} must fail"
493+
);
494+
}
495+
}
496+
462497
#[test]
463498
fn stevec_entry_term_is_optional_and_op_only() {
464499
use eql_bindings::v3::json::SteVecEntry;
@@ -524,6 +559,7 @@ fn stevec_document_and_query_schemas_are_strict() {
524559
},
525560
h: "kh".to_string().into(),
526561
sv: vec![],
562+
a: None,
527563
};
528564
let sdoc = serde_json::to_value(doc.schema()).unwrap();
529565
assert_eq!(sdoc.pointer("/additionalProperties"), Some(&json!(false)));
@@ -541,6 +577,26 @@ fn stevec_document_and_query_schemas_are_strict() {
541577
Some(&json!("#/$defs/SteVecForm"))
542578
);
543579
assert_eq!(sdoc.pointer("/$defs/SteVecForm/const"), Some(&json!("sv")));
580+
// Root `a` is optional (not required) and non-nullable, matching the SQL
581+
// CHECKs (`jsonb_typeof(val->'a') = 'boolean'`); same for the entry's `a`.
582+
assert_eq!(
583+
sdoc.pointer("/properties/a/type"),
584+
Some(&json!("boolean")),
585+
"root a must be a plain boolean, not nullable"
586+
);
587+
assert!(
588+
!sdoc["required"].as_array().unwrap().contains(&json!("a")),
589+
"root a must not be required"
590+
);
591+
let entry: eql_bindings::v3::json::SteVecEntry =
592+
serde_json::from_value(json!({ "s": "sel", "c": "ct" })).unwrap();
593+
let sentry = serde_json::to_value(entry.schema()).unwrap();
594+
assert_eq!(
595+
sentry.pointer("/properties/a/type"),
596+
Some(&json!("boolean")),
597+
"entry a must be a plain boolean, not nullable"
598+
);
599+
assert!(!sentry["required"].as_array().unwrap().contains(&json!("a")));
544600
let q = SteVecQuery { sv: vec![] };
545601
let sq = serde_json::to_value(q.schema()).unwrap();
546602
assert_eq!(sq.pointer("/additionalProperties"), Some(&json!(false)));
@@ -592,15 +648,15 @@ fn stevec_ts_exports_have_expected_shape() {
592648
};
593649

594650
// SteVecEntry: direct fields s/c, optional op and extracted metadata, and
595-
// the OPTIONAL nullable array marker `a`. `a?: boolean | null` (not `a: boolean |
596-
// null`) pins ts-rs optionality so the TS binding agrees with the JSON Schema,
651+
// the OPTIONAL, non-nullable array marker `a`. `a?: boolean` (not `a: boolean`)
652+
// pins ts-rs optionality so the TS binding agrees with the JSON Schema,
597653
// which excludes `a` from `required` — the drift a bare `Option<bool>` without
598-
// `#[ts(optional = nullable)]` silently reintroduces.
654+
// `#[ts(optional)]` silently reintroduces.
599655
let entry = export_body("SteVecEntry.ts", "SteVecEntry");
600656
for needle in [
601657
"s: Selector",
602658
"c: EntryCiphertext",
603-
"a?: boolean | null",
659+
"a?: boolean",
604660
"op?: OpeCllw",
605661
"i?: Identifier",
606662
"v?: SchemaVersion",
@@ -617,14 +673,14 @@ fn stevec_ts_exports_have_expected_shape() {
617673
// in `json.rs` (which changes the wire/consumer contract) fails here rather
618674
// than escaping to a manual diff.
619675
assert!(
620-
entry.contains("{ s: Selector, c: EntryCiphertext, a?: boolean | null, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, }"),
676+
entry.contains("{ s: Selector, c: EntryCiphertext, a?: boolean, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, }"),
621677
"SteVecEntry.ts field order must be s, c, a, op, i, v, h; got: {entry}"
622678
);
623679
let document = export_body("SteVecDocument.ts", "SteVecDocument");
624680
assert!(
625681
document.contains(
626-
"{ v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>, }"
682+
"{ v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>, a?: boolean, }"
627683
),
628-
"SteVecDocument.ts field order must be v, k, i, h, sv, got: {document}"
684+
"SteVecDocument.ts field order must be v, k, i, h, sv, a, got: {document}"
629685
);
630686
}

‎packages/eql/packages/eql/src/generated/schema/v3/eql_v3_json_entry.json‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,10 @@
4444
"$id": "https://schemas.cipherstash.com/eql/v3/eql_v3_json_entry.json",
4545
"$schema": "https://json-schema.org/draft/2020-12/schema",
4646
"additionalProperties": false,
47-
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(emitted only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
47+
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(a boolean; producers emit it only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
4848
"properties": {
4949
"a": {
50-
"type": [
51-
"boolean",
52-
"null"
53-
]
50+
"type": "boolean"
5451
},
5552
"c": {
5653
"$ref": "#/$defs/EntryCiphertext"

0 commit comments

Comments
 (0)