Skip to content

Commit 208b117

Browse files
authored
Merge pull request #1519 from cipherstash/test/protect-upgrade-no-reencryption
feat(e2e): verify and document Protect v2 to Stack v3 migrations for both stores
2 parents 2f9c8d7 + c549782 commit 208b117

28 files changed

Lines changed: 2247 additions & 104 deletions
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'stash': patch
3+
---
4+
5+
The `stash-dynamodb` skill's query examples now narrow the `encryptQuery` result before reading `hm`. The previous `result.data.hm` did not type-check, because the result type also covers ciphertexts, strings and `null`.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'stash': patch
3+
---
4+
5+
The `stash-dynamodb` skill now scopes the no-re-encryption upgrade to the tested top-level scalar equality columns and unindexed text written by `@cipherstash/protect-dynamodb@6.0.1`. It explains boolean lookup changes and the privacy cost of a plaintext replacement, stale `__hmac` removal, `downcase` normalization, and the required legacy JSON backfill with its rollback limit. It also clarifies legacy v2 reads and queries using current v3 schemas.

‎.github/workflows/tests.yml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,9 +458,11 @@ jobs:
458458
# nothing in `e2e/tests/**` loads the WASM build (the Deno smoke test that
459459
# does lives in `e2e/wasm/` and runs in `wasm-e2e-tests`).
460460
- name: Build the protect-ffi binding
461+
id: ffi-binding
461462
uses: ./.github/actions/build-ffi-binding
462463

463464
- name: Build the @cipherstash/auth binding
465+
id: auth-binding
464466
uses: ./.github/actions/build-auth-binding
465467

466468
# Run the standalone `e2e/` workspace via turbo so the `^build`
@@ -480,6 +482,65 @@ jobs:
480482
- name: Run the protect-dynamodb compatibility suite
481483
run: pnpm exec turbo run test:protect-compat --filter @cipherstash/e2e --env-mode=loose
482484

485+
# The DynamoDB <-> Postgres cross-store suite moves protect-dynamodb
486+
# 6.0.1 items into EQL v3 columns and back, so it needs a database.
487+
# `integration-db` gives this job its own compose project and ephemeral
488+
# port; the suite installs EQL v3 itself through the real
489+
# `stash eql install`. Loose env mode for the same reason as above.
490+
# The suite is independent of the preceding test suites, but still
491+
# requires both native bindings. A failed build must not start the DB
492+
# and produce a second set of missing-binding errors.
493+
- name: Start postgres
494+
id: db
495+
if: ${{ !cancelled() && steps.ffi-binding.outcome == 'success' && steps.auth-binding.outcome == 'success' }}
496+
uses: ./.github/actions/integration-db
497+
with:
498+
db: postgres
499+
500+
- name: Run the DynamoDB <-> Postgres cross-store suite
501+
if: ${{ !cancelled() && steps.db.outcome == 'success' }}
502+
run: pnpm exec turbo run test:protect-crossover --filter @cipherstash/e2e --env-mode=loose
503+
env:
504+
DATABASE_URL: ${{ steps.db.outputs.database-url }}
505+
506+
# Guarded on the project being set: if the stack never came up there is
507+
# nothing to tear down, and an unguarded `-p ""` would mask the real error.
508+
- name: Stop postgres
509+
if: always() && env.CS_COMPOSE_PROJECT != ''
510+
run: docker compose -p "$CS_COMPOSE_PROJECT" -f "$CS_COMPOSE_FILE" down -v
511+
512+
# The integration image above is PostgreSQL 17. PostgreSQL 18 records
513+
# NOT NULL constraints in pg_constraint, including the one on a primary
514+
# key's column, so the upgrade example's dependency preflight sees a
515+
# different catalogue there. No postgres-eql image exists for 18; the
516+
# upgrade suite installs EQL v2.2.1 and v3 itself, so the official image
517+
# (pinned by digest) is enough.
518+
- name: Start PostgreSQL 18
519+
id: db18
520+
if: ${{ !cancelled() && steps.ffi-binding.outcome == 'success' && steps.auth-binding.outcome == 'success' }}
521+
run: |
522+
docker run -d --name pg18-upgrade -e POSTGRES_PASSWORD=password \
523+
-p 127.0.0.1::5432 \
524+
postgres:18@sha256:e3d24f6906d0760bf4da0dae872415391ededefad23fd471c47dc53b0ddf3c42
525+
# Probe over TCP: the init-time server listens on the socket only.
526+
for _ in $(seq 1 60); do
527+
docker exec pg18-upgrade pg_isready -h 127.0.0.1 -U postgres -q && break
528+
sleep 1
529+
done
530+
docker exec pg18-upgrade pg_isready -h 127.0.0.1 -U postgres
531+
port="$(docker port pg18-upgrade 5432/tcp | head -n1 | sed 's/.*://')"
532+
echo "database-url=postgres://postgres:password@127.0.0.1:${port}/postgres" >> "$GITHUB_OUTPUT"
533+
534+
- name: Run the Postgres upgrade example on PostgreSQL 18
535+
if: ${{ !cancelled() && steps.db18.outcome == 'success' }}
536+
run: pnpm exec turbo run test:protect-crossover --filter @cipherstash/e2e --env-mode=loose -- tests/protect-postgres-upgrade.e2e.test.ts
537+
env:
538+
DATABASE_URL: ${{ steps.db18.outputs.database-url }}
539+
540+
- name: Stop PostgreSQL 18
541+
if: always() && steps.db18.outcome != 'skipped'
542+
run: docker rm -f pg18-upgrade || true
543+
483544
# Verifies @cipherstash/stack/wasm-inline works under Deno — i.e. the
484545
# WASM build of protect-ffi 0.26+ and auth 0.40+ can round-trip an
485546
# encryption against ZeroKMS / CTS in a runtime with no native

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l
9494
- `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do.
9595
- `languages/golang`: The Go module — the SDK `encrypt` with its generated-code support `encrypt/gensupport` and the policy packages `encrypt/policy` and `encrypt/policy/protosource`; the credential package `auth`; the generator `stashgen` and its command `cmd/stashgen`; and `internal` (the shared guest plumbing and the `record` wire model). A wazero host with no cgo. Its two WASI guests (`encrypt/guest`, `auth/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; `mise run wasm:guest:build:deterministic` builds the seeded TEST build the hermetic Go tests use; the `.wasm` files they embed are gitignored. The stack-encrypt guest has a second build with the EQL types (`mise run wasm:guest:build:eql`, cargo feature `eql`, linking `eql-bindings` by path; `wasm:guest:build:eql:deterministic` is its test build), embedded by `encrypt/eql` and registered on import, so a program whose generated code names an EQL type runs it (ADR-0007, amended). `encrypt/eql`'s types and `Types` table are generated from the EQL catalog by `eql-codegen` (`eql_gen.go`, written by `mise run types:generate` in `packages/eql` and drift-gated by `types:check` and `cargo test -p eql-codegen`). Generated `*_stash.go` files are committed and CI fails when `go generate ./...` changes one. Releases are cut by hand with `.github/workflows/release-golang.yml` (`gh workflow run release-golang.yml -f version=X.Y.Z[-pre] --ref <branch>`): because the guests are gitignored on main and Go fetches a version as the files committed at its tag, the workflow builds them, commits them on a release-only commit no branch points at, tags it `languages/golang/vX.Y.Z` and pushes the tag alone, then attaches the EQL v4 SQL bundle to the GitHub release. The ref must carry the EQL v4 bundle, or the build stops at "Build the EQL SQL bundles". `scripts/go-release-version.mjs` refuses, before anything is built, every version Go would refuse (non-canonical semver, `+build` metadata, a v2+ major on a module path with no `/vN` suffix, a version shaped like a pseudo-version), and a final version dispatched from anywhere but `main`; a pre-release may come from a branch. The build job also refuses a version the Go proxy already lists in `@v/list`, which catches a tag deleted after the proxy fetched it; it never asks the proxy for the version itself before the tag exists, because the proxy would cache "not found" for up to 30 minutes and `verify` would fail after the push. The build checks the `Cargo.lock` of the EQL workspace and both guest workspaces with `--locked` first, since none of those builds passes it. It runs in a job with no write token and hands the guests to a `publish` job that holds `contents: write` and runs only git and gh. `publish` fetches the commit with plain unauthenticated git rather than `actions/checkout`, whose `token` input defaults to the job's writable token, and refuses a release commit whose tree is not the one the build tested. `publish` takes the paths it commits from the workflow's fixed `GUESTS` list, never from the artifact, and refuses a `guests.sha256` that is not exactly one SHA-256 line for each of those three guests: the build job runs third-party code after writing that manifest, so a path read from it could put any file on the tag (`scripts/__tests__/go-release-artifact.test.mjs`). A version is immutable once the Go proxy has fetched it, so a tag is never moved or deleted: if `publish` or `verify` fails after the tag is pushed, use "Re-run failed jobs" on the same run, which keeps a tag that run pushed and creates or completes the release; a new dispatch of the same version is refused. Never tag a main commit by hand: `go get` would fetch a module with no guests.
9696
- `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README)
97+
- `e2e/migration-guide`: Unpublished, separately installed npm examples for the Protect/EQL v2 to Stack/EQL v3 guide. See its README before running; Postgres requires a disposable database and DynamoDB persistence is in memory.
9798
- `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker)
9899
- `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo).
99100
- `skills/*`: Agent skills (`stash-cli`, `stash-encryption`, `stash-indexing`, `stash-deployment`, `stash-zerokms`, `stash-auth`, `stash-postgres`, `stash-edge`, `stash-drizzle`, `stash-dynamodb`, `stash-supabase`, `stash-prisma`, `stash-managed-platforms`, `stash-supply-chain-security`)

‎SECURITY.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,12 @@ a fake key source, executes the Rustdoc text encryption example, and provides
5151
optional disposable PostgreSQL coverage. It adds no published package or
5252
production service.
5353

54+
`e2e/migration-guide` is an unpublished npm example package, outside the
55+
workspace release pipeline. It uses live encryption credentials and a disposable
56+
Postgres database; its DynamoDB example persists items only in memory. It is
57+
not a production migration service. Reports of unsafe migration guidance are
58+
in scope as documentation/code-example issues.
59+
5460
> **Note on publishing.** Every package in the table above, including all
5561
> seven `@cipherstash/protect-ffi*` packages, all seven `@cipherstash/auth*`
5662
> packages and `@cipherstash/eql`, is published from this repository by

0 commit comments

Comments
 (0)