Skip to content

Commit ec9c3de

Browse files
committed
test(eql): equal-term timing class, and report a failed database probe
Adds an equal-term class to the ORE timing harness (a single term against itself, against a first-block difference) and records a second machine's results, including a small C-level difference in the prefix and NULL kinds there and the cost of hashing equal terms. run.sh now tells an unreachable database apart from an existing ore_timing schema. The README says who can already run the comparator, scopes the operation-count claim to operands of the same shape, and gives sample counts per kind rather than per class.
1 parent 8aa3db0 commit ec9c3de

3 files changed

Lines changed: 63 additions & 27 deletions

File tree

‎packages/eql/tests/timing/ore_block_256/README.md‎

Lines changed: 39 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,14 @@
22

33
Checks that `eql_v3_internal.compare_ore_block_256_terms`, the comparator
44
behind every ordered ORE operator and the ORE operator class, takes the same
5-
time wherever its operands first differ.
5+
time wherever its operands first differ, and whether they differ at all.
66

77
## Why it matters
88

9-
The comparator's inputs are ciphertexts, so the database operator learns
10-
nothing from its timing: they can run the comparison themselves. The party
11-
who gains is someone who can issue queries and time them without reading the
9+
The database operator gains nothing from the comparator's timing. They hold
10+
the stored ciphertexts and can run the comparator on any two of them
11+
directly, which tells them everything its timing could. The party who gains
12+
is someone who can issue queries and time them without reading the
1213
stored ciphertexts, such as an application user. Query results already tell
1314
them the order of their value relative to stored ones. If the comparison time
1415
depends on where the two values first differ, timing also tells them how long
@@ -17,9 +18,12 @@ comparison when the first of six terms differs, 47 µs when only the sixth
1718
does.
1819

1920
PL/pgSQL is not a constant-time environment. What it can give is a constant
20-
*operation count*: the same statements and function calls on every path, so
21-
nothing at the scale of a statement (fractions of a microsecond) depends on
22-
the data. C-level effects inside those statements (`memcmp` stopping early
21+
*operation count*: for operands of the same shape, the same statements and
22+
function calls whatever their contents, so nothing at the scale of a
23+
statement (fractions of a microsecond) depends on where or whether they
24+
differ. The shape is the term count, the term width and the position of any
25+
NULL element. It is public structure, and the comparator may branch on it:
26+
a single term and a six-term value take different paths. C-level effects inside those statements (`memcmp` stopping early
2327
within 16 bytes, the offset a `substr` reads at) remain, orders of magnitude
2428
smaller. The Rust comparator in ore.rs is constant-time at the instruction
2529
level; this is the PL/pgSQL counterpart.
@@ -51,6 +55,8 @@ plus a correctness set:
5155

5256
- **single term**: pairs differing in the first block against pairs differing
5357
only in the last (block 7);
58+
- **equal term**: a single term against itself, against pairs differing in
59+
the first block;
5460
- **six terms** (text-shaped values): pairs differing in the first term
5561
against pairs differing only in the sixth;
5662
- **prefix**: five terms against six, where the five equal the first five of
@@ -81,21 +87,38 @@ difference.
8187
## Results
8288

8389
PostgreSQL 17.11, aarch64 (Docker on an Apple M1 Max). Welch t for two runs
84-
each; the current comparator at 200,000 single-term samples of 16 comparisons
85-
and 50,000 multi-term samples of 8 per class:
90+
each; the current comparator at 200,000 samples of 16 comparisons for the
91+
single-term kind and 50,000 samples of 8 for each multi-term kind, each split
92+
at random between the kind's two classes:
8693

87-
| comparator | single term | six terms | prefix | NULL element |
88-
|---|---:|---:|---:|---:|
89-
| original (term by term) | +4.9, +13.9 | −1552, −2575 | +517, +459 ¹ | +520, +682 ¹ |
90-
| first constant-count version ² | +1.8, +1.1 | +4.5, +2.7 | −62, −69 ³ | −60, −70 ³ |
91-
| current | +2.24, +0.15 | +1.03, −0.27 | −0.68, +0.18 | −0.57, +0.14 |
94+
| comparator | single term | equal term | six terms | prefix | NULL element |
95+
|---|---:|---:|---:|---:|---:|
96+
| original (term by term) | +4.9, +13.9 | | −1552, −2575 | +517, +459 ¹ | +520, +682 ¹ |
97+
| first constant-count version ² | +1.8, +1.1 | | +4.5, +2.7 | −62, −69 ³ | −60, −70 ³ |
98+
| current | +2.24, +0.15 | | +1.03, −0.27 | −0.68, +0.18 | −0.57, +0.14 |
99+
| current, second machine ⁴ | +1.60, +0.71 | −2.49, −1.99 | −2.58, +1.33 | −4.96, −1.19, −2.56, −7.42, −5.14 | −3.82, −9.63, −2.23, −3.88, −2.88 |
92100

93101
¹ 5,000 samples per run. ² It scanned every block of every term, and skipped
94102
its hash when the term counts or a NULL settled the order. ³ 10,000 samples
95-
per run.
103+
per run. ⁴ PostgreSQL 17.5 run natively on macOS, Apple silicon, at the same
104+
sample counts, with three more runs of the prefix and NULL kinds.
105+
106+
On the second machine the prefix and NULL-element kinds come out negative on
107+
every run, and three runs pass |t| = 5. Class 0, where the structure settles
108+
the order, is faster by 0.1 to 0.2 µs per sample of eight comparisons: about
109+
20 ns a comparison, or 0.2% of one. That is below the cost of a single
110+
statement, so it is a C-level effect of the kind described above, not a
111+
skipped statement; a skipped hash showed up as t = −60 to −70 at a tenth of
112+
the samples. The equal-term class shows the same sign at about 6 ns a
113+
comparison.
96114

97115
Cost: the current comparator takes about 6% more than the original per
98-
single-term comparison, and about 18 µs per six-term comparison whatever the
116+
single-term comparison when the terms differ, and about 36% more when they are
117+
equal (3.6 µs against 2.7 µs on the second machine, calling
118+
`compare_ore_block_256_term` directly), because equal terms now run the hash
119+
too. Duplicates are common in index builds, `ORDER BY` ties and equality
120+
probes on ORE columns, so expect ordered workloads with many ties to see the
121+
larger figure. It takes about 18 µs per six-term comparison whatever the
99122
operands. The original took 11 µs when the first term differs and 47 µs when
100123
only the sixth does; the first constant-count version took 31 µs.
101124

‎packages/eql/tests/timing/ore_block_256/harness.sql‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -24,17 +24,21 @@ CREATE TABLE pairs AS
2424
(label = 'last')::int AS cls, terms_of(a_hex) AS a, terms_of(b_hex) AS b
2525
FROM staging WHERE kind LIKE 'timing%';
2626

27-
-- Two shapes built from the six-term pairs, where a and b differ in term 1.
28-
-- Class 1 in both is a's first five terms against b (term 1 differs). Class 0:
27+
-- Three shapes built from the pairs that differ in the first block (single
28+
-- terms) or the first term (six terms). Class 1 is such a difference in all
29+
-- three. Class 0:
30+
-- equal: a single term against itself: no block differs;
2931
-- prefix: a's first five terms against all six of a: every paired term is
3032
-- equal and the term counts differ;
3133
-- null: a's first five terms and a NULL against a: the first unequal pair
3234
-- is the NULL one, after five equal terms.
33-
-- A comparator that skips its hash once the order is settled by structure
34-
-- shows a difference here.
35+
-- A comparator that skips its hash for equal terms, or once the order is
36+
-- settled by structure, shows a difference here.
3537
INSERT INTO pairs
3638
SELECT kind, row_number() OVER (PARTITION BY kind ORDER BY random()), cls, a, b FROM (
37-
SELECT 'prefix' AS kind, 0 AS cls, a[1:5] AS a, a AS b FROM pairs WHERE kind = 'timing6' AND cls = 0
39+
SELECT 'equal' AS kind, 0 AS cls, a, a AS b FROM pairs WHERE kind = 'timing1' AND cls = 0
40+
UNION ALL SELECT 'equal', 1, a, b FROM pairs WHERE kind = 'timing1' AND cls = 0
41+
UNION ALL SELECT 'prefix', 0, a[1:5], a FROM pairs WHERE kind = 'timing6' AND cls = 0
3842
UNION ALL SELECT 'prefix', 1, a[1:5], b FROM pairs WHERE kind = 'timing6' AND cls = 0
3943
UNION ALL SELECT 'null', 0, a[1:5] || NULL::eql_v3_internal.ore_block_256_term, a
4044
FROM pairs WHERE kind = 'timing6' AND cls = 0
@@ -91,8 +95,8 @@ $$ LANGUAGE plpgsql;
9195
-- Report this, not a percentile-cropped maximum: cropping turns a difference
9296
-- in spread into a large t with no difference in mean.
9397
CREATE FUNCTION welch(label text, tbl regclass)
94-
RETURNS TABLE (run text, n bigint, mean_first numeric, mean_last numeric,
95-
median_first numeric, median_last numeric, t numeric)
98+
RETURNS TABLE (run text, n bigint, mean_0 numeric, mean_1 numeric,
99+
median_0 numeric, median_1 numeric, t numeric)
96100
AS $$
97101
BEGIN
98102
RETURN QUERY EXECUTE format($q$

‎packages/eql/tests/timing/ore_block_256/run.sh‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@
66
#
77
# DATABASE_URL must point at a database with EQL installed (e.g. the release
88
# installer from `mise run build`). Needs psql and cargo. samples defaults to
9-
# 200000 single-term samples (and a quarter as many six-term ones), a few
10-
# minutes in all. See README.md for the method and how to read the output.
9+
# 200000 samples per single-term kind (and a quarter as many per multi-term
10+
# kind), a few minutes in all. See README.md for the method and how to read
11+
# the output.
1112
set -euo pipefail
1213

1314
samples="${1:-200000}"
@@ -16,8 +17,13 @@ here="$(cd "$(dirname "$0")" && pwd)"
1617

1718
# Everything lives in the ore_timing schema, which is dropped on exit. Refuse
1819
# to start if it already exists, so a run never drops a schema it did not make.
19-
if [ "$(psql "$DATABASE_URL" -tA -c "SELECT to_regnamespace('ore_timing') IS NOT NULL")" != "f" ]; then
20-
echo "schema ore_timing already exists (or the database is unreachable); not touching it" >&2
20+
# A failed psql is reported as itself, not as an existing schema.
21+
if ! exists="$(psql "$DATABASE_URL" -tA -c "SELECT to_regnamespace('ore_timing') IS NOT NULL")"; then
22+
echo "could not query the database at DATABASE_URL (see the psql error above)" >&2
23+
exit 1
24+
fi
25+
if [ "$exists" != "f" ]; then
26+
echo "schema ore_timing already exists; not touching it" >&2
2127
exit 1
2228
fi
2329

@@ -53,13 +59,16 @@ echo "== timing (two runs per kind; classes: see README.md)"
5359
for run in 1 2; do
5460
psql "$DATABASE_URL" -q -v ON_ERROR_STOP=1 -c "
5561
CREATE TABLE ore_timing.single_$run AS SELECT * FROM ore_timing.bench('timing1', $samples, 16);
62+
CREATE TABLE ore_timing.equal_$run AS SELECT * FROM ore_timing.bench('equal', $samples, 16);
5663
CREATE TABLE ore_timing.six_$run AS SELECT * FROM ore_timing.bench('timing6', $((samples / 4)), 8);
5764
CREATE TABLE ore_timing.prefix_$run AS SELECT * FROM ore_timing.bench('prefix', $((samples / 4)), 8);
5865
CREATE TABLE ore_timing.null_$run AS SELECT * FROM ore_timing.bench('null', $((samples / 4)), 8);"
5966
done
6067
psql "$DATABASE_URL" -c "
6168
SELECT * FROM ore_timing.welch('single term, 16/sample #1', 'ore_timing.single_1')
6269
UNION ALL SELECT * FROM ore_timing.welch('single term, 16/sample #2', 'ore_timing.single_2')
70+
UNION ALL SELECT * FROM ore_timing.welch('equal term, 16/sample #1', 'ore_timing.equal_1')
71+
UNION ALL SELECT * FROM ore_timing.welch('equal term, 16/sample #2', 'ore_timing.equal_2')
6372
UNION ALL SELECT * FROM ore_timing.welch('six terms, 8/sample #1', 'ore_timing.six_1')
6473
UNION ALL SELECT * FROM ore_timing.welch('six terms, 8/sample #2', 'ore_timing.six_2')
6574
UNION ALL SELECT * FROM ore_timing.welch('prefix, 8/sample #1', 'ore_timing.prefix_1')

0 commit comments

Comments
 (0)