Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4e067bd
feat(migrate): shared chunked engine and reverse (decrypt) backfill
calvinbrewer Oct 13, 2026
dabc84a
refactor(cli): share migration-file writing, table output and PK dete…
calvinbrewer Oct 13, 2026
531d22a
feat(cli): add stash decrypt plan/backfill/drop/status for reverse mi…
calvinbrewer Oct 13, 2026
bca02b6
docs(cli): document stash decrypt and extend the encrypt e2e to a rou…
calvinbrewer Oct 13, 2026
1001ff7
fix(migrate): keep decrypted plaintext out of write errors and resume…
calvinbrewer Oct 14, 2026
7bfc234
fix(cli): refuse stale plaintext twins and keep decrypt dual-writes u…
calvinbrewer Oct 14, 2026
c332f23
fix(cli): derive the decrypt status twin name with the rules plan uses
calvinbrewer Oct 14, 2026
23a157e
docs(skills): keep dual-writing until the decrypt drop applies, and d…
calvinbrewer Oct 14, 2026
106ea90
fix(migrate): keep a decrypt pass in the mode it started
calvinbrewer Oct 14, 2026
5429a39
fix(migrate): finish a started forced decrypt pass on every later run
tobyhede Oct 15, 2026
c1fc1b9
fix(migrate): owe a forced pass from the --force acknowledgement of a…
tobyhede Oct 15, 2026
d0b4ac9
test(migrate): pin that a completed forced pass closes the obligation
tobyhede Oct 15, 2026
57b94b0
fix(cli): refuse to reverse a forward encryption that was never dropped
tobyhede Oct 15, 2026
597089b
fix(migrate): guard decrypt writes against concurrent dual-writes
tobyhede Oct 16, 2026
3ff6002
fix(cli): close a forward migration whose source was removed outside …
tobyhede Oct 16, 2026
b56561f
fix(migrate,cli): clear skipped rows in a forced decrypt pass
claude Oct 20, 2026
2ab2b94
docs(cli,migrate): warn that a forced decrypt pass clears undecryptab…
claude Oct 20, 2026
eb8194b
fix(cli): stop telling users to hand-edit .cipherstash/migrations.json
claude Oct 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/stash-decrypt-reverse-migration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'stash': minor
'@cipherstash/migrate': minor
---

`stash decrypt status|plan|backfill|drop` moves encrypted columns back to plaintext — the encrypt lifecycle run in reverse, with the same safety rails. `decrypt plan` adds a nullable plaintext twin per EQL v3 column with its type mapped back from the domain (`text*` → `text`, `timestamp*` → `timestamptz`, `json*` → `jsonb`, …): Drizzle schema lines for `drizzle-kit generate`, or a reviewed SQL migration elsewhere. `decrypt backfill` decrypts in resumable, checkpointed chunks, one `bulkDecrypt` (one ZeroKMS request) per chunk, default 500 rows. Values bound to an end-user lock context cannot be decrypted with workspace credentials; they are skipped, counted, left NULL and reported, and `decrypt drop` refuses while any remain. `decrypt drop` re-verifies coverage at apply time, drops the indexes built on the column's EQL terms, drops the column, and with `--uninstall-eql` removes EQL v3 when it was the last encrypted column in the database.

`@cipherstash/migrate`: `runBackfill` now runs on a shared chunked engine (`runChunkedMigration`), which also drives the new `runDecryptBackfill`. Forward behaviour and recorded event details are unchanged. Adds `countUndecrypted`, `fetchUndecryptedPage`, `plaintextTypeForEqlDomain`, `columnSqlType`, `listColumnIndexes`, `countEqlColumnsInDatabase`, `removeManifestColumn`, and the reverse event/phase vocabulary (`decrypt_started`, `decrypt_checkpoint`, `decrypted`, `encrypted_dropped`; `decrypt-dual-writing`, `decrypting`, `decrypted`, `encrypted-dropped`), recorded against the encrypted column. `MigrationPhase` and `MigrationEvent` gain those members, so an exhaustive `switch` over them needs the new cases.
33 changes: 31 additions & 2 deletions languages/typescript/packages/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ npx stash auth login
└── npx stash status ← where am I?
```

`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, and the staged EQL v3 encryption rollout.
`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, the staged EQL v3 encryption rollout, and its reverse: moving encrypted columns back to plaintext (`stash decrypt`).

---

Expand All @@ -66,7 +66,7 @@ export default defineConfig({

The CLI loads `.env` files automatically before reading the config, so `process.env` references work without extra setup. The config file is resolved by walking up from the current working directory.

Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, and `encrypt *`.
Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, `encrypt *`, and `decrypt *`.

---

Expand Down Expand Up @@ -310,6 +310,35 @@ Reads `databaseUrl` from `stash.config.ts`.

---

### `npx stash decrypt` — moving columns back to plaintext

Leaving CipherStash is the encrypt lifecycle run backwards, with the same safety rails. Decryption is never charged.

```
decrypt plan add a plaintext twin per encrypted column (type mapped back from its EQL domain)
→ deploy dual-writes (app writes the encrypted column AND its plaintext twin)
decrypt backfill decrypt every row into the twin — resumable, one ZeroKMS request per chunk
→ switch reads to the plaintext column and deploy
decrypt drop generate the migration that drops the encrypted column and its EQL indexes
decrypt status where each column stands
```

```bash
npx stash decrypt plan # read-only overview
npx stash decrypt plan --table users # generate the twins
npx stash decrypt backfill --table users --column email_encrypted
npx stash decrypt drop --table users --column email_encrypted [--uninstall-eql]
npx stash decrypt status
```

`--column` always names the column being migrated away from — the encrypted one here. The plaintext twin defaults to the original name recorded in `.cipherstash/migrations.json`, else the column name without `_encrypted`; override with `--plaintext-column`. Plaintext types: `text*` → `text`, `integer*`/`smallint*`/`bigint*`/`numeric*`/`real*` → the same type, `double*` → `double precision`, `boolean` → `boolean`, `date*` → `date`, `timestamp*` → `timestamptz` (EQL timestamps are instants), `json`/`json_search` → `jsonb`. The backfill casts to whatever type the twin actually has, so you can widen or narrow it in the migration.

**Values bound to a lock context are not decrypted.** A value encrypted with an end-user lock context can only be decrypted as that user. The backfill skips those rows, counts them, leaves the twin NULL and reports them; `decrypt drop` refuses while any remain, because dropping the encrypted column would destroy their only copy. Decrypt them in your application on each user's behalf, then re-run the backfill.

The backfill loads your encryption client exactly as `encrypt backfill` does (client file or Prisma Next contract), so export the `CS_*` credentials of the environment that wrote the data.

---

## Migration mode

Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema.
Expand Down
52 changes: 51 additions & 1 deletion languages/typescript/packages/cli/scripts/e2e-encrypt.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/usr/bin/env bash
# End-to-end smoke test for `stash encrypt`.
# End-to-end smoke test for `stash encrypt`, then `stash decrypt` back to
# plaintext — the full round trip, verified against the original values.
#
# Requires a local Postgres you have superuser on. Creates & destroys
# `stash_e2e_test`. Requires CipherStash credentials in the environment
Expand Down Expand Up @@ -54,5 +55,54 @@ echo "==> 5. Switch application reads to email_encrypted (manual deploy step)"

echo "==> 6. Generate plaintext drop migration"
"$STASH" encrypt drop --table users --column email --migrations-dir "$(pwd)/drizzle"
DROP_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_drop_users_email.sql | head -1)
psql "$DATABASE_URL" -f "$DROP_MIGRATION" >/dev/null
ORIGINAL=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg('user-' || id || '@example.com', ',' ORDER BY id)) FROM users")

# ---------------------------------------------------------------------------
# Reverse lifecycle: move the column back to plaintext (stash decrypt).
# ---------------------------------------------------------------------------

echo "==> 7. Plan the plaintext twin (users.email_encrypted → email text)"
"$STASH" decrypt plan
"$STASH" decrypt plan --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle"
ADD_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_add_plaintext_users_email.sql | head -1)
psql "$DATABASE_URL" -f "$ADD_MIGRATION" >/dev/null

echo "==> 8. Decrypt backfill with interrupt/resume (reverse dual-writes confirmed via flag)"
"$STASH" decrypt backfill --table users --column email_encrypted --chunk-size 500 --confirm-dual-writes-deployed &
PID=$!
sleep 2
kill -INT "$PID" || true
wait "$PID" || true
"$STASH" decrypt backfill --table users --column email_encrypted

REMAINING=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM users WHERE email_encrypted IS NOT NULL AND email IS NULL")
if [ "$REMAINING" != "0" ]; then
echo "FAIL: ${REMAINING} rows still undecrypted" >&2
exit 1
fi
ROUNDTRIP=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg(email, ',' ORDER BY id)) FROM users")
if [ "$ROUNDTRIP" != "$ORIGINAL" ]; then
echo "FAIL: decrypted plaintext does not match the original values" >&2
exit 1
fi
echo "OK: all 5000 rows decrypted back to their original plaintext"

echo "==> 9. Reverse status"
"$STASH" decrypt status

echo "==> 10. Switch application reads to email (manual deploy step)"

echo "==> 11. Generate and apply the encrypted-column drop"
"$STASH" decrypt drop --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle"
DECRYPT_DROP=$(ls -t "$(pwd)/drizzle"/*_drop_encrypted_users_email_encrypted.sql | head -1)
psql "$DATABASE_URL" -f "$DECRYPT_DROP" >/dev/null
GONE=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'users' AND column_name = 'email_encrypted'")
if [ "$GONE" != "0" ]; then
echo "FAIL: email_encrypted still exists" >&2
exit 1
fi
echo "OK: users is plaintext again"

echo "==> Done."
77 changes: 77 additions & 0 deletions languages/typescript/packages/cli/src/bin/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,11 @@ Commands:
encrypt backfill Resumably encrypt plaintext into the encrypted column
encrypt drop Generate a migration to drop the plaintext column

decrypt status Show per-column reverse-migration status (phase, progress, undecryptable)
decrypt plan Plan (or generate) the plaintext twins for encrypted columns
decrypt backfill Resumably decrypt ciphertext into the plaintext column
decrypt drop Generate a migration to drop the encrypted column

env Mint deployment credentials and print them as env vars

Options:
Expand Down Expand Up @@ -466,6 +471,75 @@ async function runEncryptCommand(
}
}

async function runDecryptCommand(
sub: string | undefined,
flags: Record<string, boolean>,
values: Record<string, string>,
) {
switch (sub) {
case 'status': {
const { decryptStatusCommand } = await requireStack(
() => import('../commands/decrypt/status.js'),
)
await decryptStatusCommand()
break
}
case 'plan': {
const { decryptPlanCommand } = await requireStack(
() => import('../commands/decrypt/plan.js'),
)
await decryptPlanCommand({
table: values.table,
column: values.column,
plaintextColumn: values['plaintext-column'],
migrationsDir: values['migrations-dir'],
dryRun: flags['dry-run'],
})
break
}
case 'backfill': {
const table = requireValue(values, 'table')
const column = requireValue(values, 'column')
const { decryptBackfillCommand } = await requireStack(
() => import('../commands/decrypt/backfill.js'),
)
await decryptBackfillCommand({
table,
column,
plaintextColumn: values['plaintext-column'],
pkColumn: values['pk-column'],
chunkSize: values['chunk-size']
? Number(values['chunk-size'])
: undefined,
confirmDualWritesDeployed: flags['confirm-dual-writes-deployed'],
force: flags.force,
continueOnUndecryptable: flags['continue-on-undecryptable'],
})
break
}
case 'drop': {
const table = requireValue(values, 'table')
const column = requireValue(values, 'column')
const { decryptDropCommand } = await requireStack(
() => import('../commands/decrypt/drop.js'),
)
await decryptDropCommand({
table,
column,
plaintextColumn: values['plaintext-column'],
migrationsDir: values['migrations-dir'],
uninstallEql: flags['uninstall-eql'],
})
break
}
default:
p.log.error(`${messages.decrypt.unknownSubcommand}: ${sub ?? '(none)'}`)
console.log()
console.log(HELP)
throw new CliExit(1)
}
}

function requireValue(values: Record<string, string>, key: string): string {
const v = values[key]
if (!v) {
Expand Down Expand Up @@ -610,6 +684,9 @@ async function dispatch(
case 'encrypt':
await runEncryptCommand(subcommand, flags, values)
break
case 'decrypt':
await runDecryptCommand(subcommand, flags, values)
break
case 'schema':
await runSchemaCommand(subcommand, flags, values)
break
Expand Down
134 changes: 134 additions & 0 deletions languages/typescript/packages/cli/src/cli/registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,12 @@ const COLUMN_FLAG: Flag = {
value: '<name>',
description: 'Target column.',
}
const PLAINTEXT_COLUMN_FLAG: Flag = {
name: '--plaintext-column',
value: '<name>',
description:
"The plaintext twin. Default: the pairing a previous run recorded, else the original name from .cipherstash/migrations.json, else the column name without its '_encrypted' suffix.",
}

/**
* The registry. Ordered as it should render in help. Populated from the existing
Expand Down Expand Up @@ -634,6 +640,134 @@ export const registry: CommandGroup[] = [
},
],
},
{
title: 'Decrypt',
commands: [
{
// Zero-arg like `encrypt status` — it reads every EQL v3 column.
name: 'decrypt status',
summary:
'Show per-column reverse-migration status (phase, progress, undecryptable)',
},
{
name: 'decrypt plan',
summary: 'Plan (or generate) the plaintext twins for encrypted columns',
long: [
'Step one of moving encrypted columns back to plaintext. With no flags,',
'lists every EQL v3 column in the current schema, the plaintext twin it',
'would reverse into (type mapped back from its EQL domain), and the next',
'command to run. With --table, produces the change that adds the twins:',
'Drizzle schema lines for drizzle-kit generate, or a reviewed SQL',
'migration elsewhere. Twins are added nullable; nothing is applied.',
].join('\n'),
examples: [
'decrypt plan',
'decrypt plan --table users',
'decrypt plan --table users --column email_encrypted --dry-run',
],
flags: [
{
...TABLE_FLAG,
description: 'Generate for this table (omit for the overview).',
},
{
...COLUMN_FLAG,
description:
'One encrypted column on --table (default: every EQL v3 column on it).',
},
PLAINTEXT_COLUMN_FLAG,
{
name: '--migrations-dir',
value: '<path>',
description:
'Directory to write the migration into (non-Drizzle projects).',
},
DRY_RUN_FLAG,
],
},
{
name: 'decrypt backfill',
summary: 'Resumably decrypt ciphertext into the plaintext column',
long: [
'The reverse of `encrypt backfill`, on the same engine: chunked,',
'resumable, idempotent, one bulkDecrypt (one ZeroKMS request) per chunk.',
'Values that cannot be decrypted with these credentials, typically ones',
'bound to an end-user lock context, are skipped, counted, left NULL and',
'reported; `decrypt drop` refuses while any remain.',
].join('\n'),
examples: [
'decrypt backfill --table users --column email_encrypted',
'decrypt backfill --table users --column email_encrypted --plaintext-column email',
],
flags: [
TABLE_FLAG,
{
...COLUMN_FLAG,
description: 'The encrypted (EQL v3) column to decrypt.',
},
PLAINTEXT_COLUMN_FLAG,
{
name: '--pk-column',
value: '<name>',
description: 'Primary-key column used to page through rows.',
},
{
name: '--chunk-size',
value: '<n>',
description: 'Rows decrypted per batch — one ZeroKMS request each.',
default: '500',
},
{
name: '--confirm-dual-writes-deployed',
description:
'Assert the app writes both the encrypted and plaintext columns (safety gate).',
},
{
name: '--force',
description:
'Re-decrypt rows whose plaintext is already set (drift recovery).',
},
{
name: '--continue-on-undecryptable',
description:
'Carry on when the first batch decrypts nothing, recording those rows as skipped.',
},
],
},
{
name: 'decrypt drop',
summary: 'Generate a migration to drop the encrypted column',
long: [
'Runs from the `decrypted` phase. Verifies live coverage (no row with',
'ciphertext set and plaintext NULL), then writes a migration that',
're-checks coverage at apply time, drops the indexes built on the',
"column's EQL terms, and drops the column. Not applied for you.",
].join('\n'),
examples: [
'decrypt drop --table users --column email_encrypted',
'decrypt drop --table users --column email_encrypted --uninstall-eql',
],
flags: [
TABLE_FLAG,
{
...COLUMN_FLAG,
description: 'The encrypted (EQL v3) column to drop.',
},
PLAINTEXT_COLUMN_FLAG,
{
name: '--migrations-dir',
value: '<path>',
description: 'Directory to write the drop migration into.',
},
{
name: '--uninstall-eql',
description:
'Also uninstall EQL v3 (its schemas and public.eql_v3_* domains) in the same migration — only when this is the last EQL v3 column in the database (re-checked at apply time).',
},
],
},
],
},
{
title: 'Deployment',
commands: [
Expand Down
Loading
Loading