Skip to content

Commit 214d24d

Browse files
authored
Merge pull request #1193 from cipherstash/cj/cip-4165-stash-cli-reverse-migration-decrypt-backfill-for-moving
feat(cli): reverse migration (stash decrypt) for moving columns back to plaintext
2 parents 6e9f1ee + eb8194b commit 214d24d

49 files changed

Lines changed: 7720 additions & 272 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
'stash': minor
3+
'@cipherstash/migrate': minor
4+
---
5+
6+
`stash decrypt status|plan|backfill|drop` moves encrypted columns back to plaintext — the encrypt lifecycle run in reverse, with the same safety rails. `decrypt plan` adds a nullable plaintext twin per EQL v3 column with its type mapped back from the domain (`text*` → `text`, `timestamp*` → `timestamptz`, `json*` → `jsonb`, …): Drizzle schema lines for `drizzle-kit generate`, or a reviewed SQL migration elsewhere. `decrypt backfill` decrypts in resumable, checkpointed chunks, one `bulkDecrypt` (one ZeroKMS request) per chunk, default 500 rows. It refuses a plaintext column that a forward encryption started from and never dropped: only one migration may be open on a pair of columns, so finish or abandon the forward one first. When that column was already removed outside stash, `decrypt plan` finds it missing and records the forward migration as dropped (marked as observed), so the reverse migration can start. Its first run also refuses a plaintext column that already holds values, because those values may be stale and the backfill only fills NULLs: pass `--force` to overwrite them from the ciphertext or `--confirm-plaintext-current` to keep them. Once `--force` is chosen, or any forced run starts, every later run is forced until a forced pass completes, so an interrupted overwrite is always finished. The application keeps writing both columns until the drop migration is applied. Values bound to an end-user lock context cannot be decrypted with workspace credentials; they are skipped, counted, left NULL and reported, and `decrypt drop` refuses while any remain. `decrypt drop` re-verifies coverage at apply time, drops the indexes built on the column's EQL terms, drops the column, and with `--uninstall-eql` removes EQL v3 when it was the last encrypted column in the database.
7+
8+
`@cipherstash/migrate`: `runBackfill` now runs on a shared chunked engine (`runChunkedMigration`), which also drives the new `runDecryptBackfill`. Forward behaviour and recorded event details are unchanged. Adds `runDecryptBackfill` (resumes after a failed run; `PlaintextWriteError` keeps decrypted values out of write errors; chunk size capped at `MAX_DECRYPT_CHUNK_SIZE`), `columnHasValues`, `countUndecrypted`, `fetchUndecryptedPage`, `plaintextTypeForEqlDomain`, `columnSqlType`, `listColumnIndexes`, `primaryKeyColumns`, `countEqlColumnsInDatabase`, `removeManifestColumn`, and the reverse event/phase vocabulary (`decrypt_started`, `decrypt_checkpoint`, `decrypted`, `encrypted_dropped`; `decrypt-dual-writing`, `decrypting`, `decrypted`, `encrypted-dropped`), recorded against the encrypted column. `MigrationPhase` and `MigrationEvent` gain those members, so an exhaustive `switch` over them needs the new cases.

‎languages/typescript/packages/cli/README.md‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ npx stash auth login
4242
└── npx stash status ← where am I?
4343
```
4444

45-
`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, and the staged EQL v3 encryption rollout.
45+
`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, the staged EQL v3 encryption rollout, and its reverse: moving encrypted columns back to plaintext (`stash decrypt`).
4646

4747
---
4848

@@ -66,7 +66,7 @@ export default defineConfig({
6666

6767
The CLI loads `.env` files automatically before reading the config, so `process.env` references work without extra setup. The config file is resolved by walking up from the current working directory.
6868

69-
Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, and `encrypt *`.
69+
Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, `encrypt *`, and `decrypt *`.
7070

7171
---
7272

@@ -310,6 +310,40 @@ Reads `databaseUrl` from `stash.config.ts`.
310310

311311
---
312312

313+
### `npx stash decrypt` — moving columns back to plaintext
314+
315+
Leaving CipherStash is the encrypt lifecycle run backwards, with the same safety rails. Decryption is never charged.
316+
317+
```
318+
decrypt plan add a plaintext twin per encrypted column (type mapped back from its EQL domain)
319+
→ deploy dual-writes (app writes the encrypted column AND its plaintext twin)
320+
decrypt backfill decrypt every row into the twin — resumable, one ZeroKMS request per chunk
321+
→ switch reads to the plaintext column and deploy (keep writing BOTH columns)
322+
decrypt drop generate the migration that drops the encrypted column and its EQL indexes
323+
→ apply it, then stop writing the encrypted column
324+
decrypt status where each column stands
325+
```
326+
327+
```bash
328+
npx stash decrypt plan # read-only overview
329+
npx stash decrypt plan --table users # generate the twins
330+
npx stash decrypt backfill --table users --column email_encrypted
331+
npx stash decrypt drop --table users --column email_encrypted [--uninstall-eql]
332+
npx stash decrypt status
333+
```
334+
335+
`--column` always names the column being migrated away from — the encrypted one here. The plaintext twin defaults to the pairing the reverse migration in progress recorded, then the original name recorded in `.cipherstash/migrations.json`, else the column name without `_encrypted`, else `<column>_plaintext`; override with `--plaintext-column`. Plaintext types: `text*` → `text`, `integer*`/`smallint*`/`bigint*`/`numeric*`/`real*` → the same type, `double*` → `double precision`, `boolean` → `boolean`, `date*` → `date`, `timestamp*` → `timestamptz` (EQL timestamps are instants), `json`/`json_search` → `jsonb`. The backfill casts to whatever type the twin actually has, so you can widen or narrow it in the migration.
336+
337+
**Keep dual-writing until the drop is applied.** Unlike the forward rollout, the application keeps writing both the encrypted column and the plaintext column until the `decrypt drop` migration has run. `decrypt backfill --force` and the drop's coverage check depend on it: if writes to the encrypted column stop early, a value cleared in the plaintext column keeps its old ciphertext, and a re-run of the backfill writes the old value back.
338+
339+
**A forward encryption that was never dropped cannot be reversed.** The backfill refuses a plaintext column that is the source of a forward encryption not yet dropped: finish it with `encrypt drop`, or abandon it by stopping writes to the encrypted column and dropping it. **A plaintext column that already holds values is refused on the first run.** The backfill only fills NULLs, so it refuses such a column until you pass `--force` (overwrite it from the ciphertext; every later run is then forced until a forced pass completes) or `--confirm-plaintext-current` (keep it; only if it was written on every change). `decrypt plan` warns about the same columns.
340+
341+
**Values bound to a lock context are not decrypted.** A value encrypted with an end-user lock context can only be decrypted as that user. The backfill skips those rows, counts them, leaves the twin NULL and reports them; `decrypt drop` refuses while any remain, because dropping the encrypted column would destroy their only copy. Decrypt them in your application on each user's behalf, then re-run the backfill.
342+
343+
The backfill loads your encryption client exactly as `encrypt backfill` does (client file or Prisma Next contract), so export the `CS_*` credentials of the environment that wrote the data.
344+
345+
---
346+
313347
## Migration mode
314348

315349
Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema.

‎languages/typescript/packages/cli/scripts/e2e-encrypt.sh‎

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
#!/usr/bin/env bash
2-
# End-to-end smoke test for `stash encrypt`.
2+
# End-to-end smoke test for `stash encrypt`, then `stash decrypt` back to
3+
# plaintext — the full round trip, verified against the original values.
34
#
45
# Requires a local Postgres you have superuser on. Creates & destroys
56
# `stash_e2e_test`. Requires CipherStash credentials in the environment
@@ -54,5 +55,54 @@ echo "==> 5. Switch application reads to email_encrypted (manual deploy step)"
5455

5556
echo "==> 6. Generate plaintext drop migration"
5657
"$STASH" encrypt drop --table users --column email --migrations-dir "$(pwd)/drizzle"
58+
DROP_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_drop_users_email.sql | head -1)
59+
psql "$DATABASE_URL" -f "$DROP_MIGRATION" >/dev/null
60+
ORIGINAL=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg('user-' || id || '@example.com', ',' ORDER BY id)) FROM users")
61+
62+
# ---------------------------------------------------------------------------
63+
# Reverse lifecycle: move the column back to plaintext (stash decrypt).
64+
# ---------------------------------------------------------------------------
65+
66+
echo "==> 7. Plan the plaintext twin (users.email_encrypted → email text)"
67+
"$STASH" decrypt plan
68+
"$STASH" decrypt plan --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle"
69+
ADD_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_add_plaintext_users_email.sql | head -1)
70+
psql "$DATABASE_URL" -f "$ADD_MIGRATION" >/dev/null
71+
72+
echo "==> 8. Decrypt backfill with interrupt/resume (reverse dual-writes confirmed via flag)"
73+
"$STASH" decrypt backfill --table users --column email_encrypted --chunk-size 500 --confirm-dual-writes-deployed &
74+
PID=$!
75+
sleep 2
76+
kill -INT "$PID" || true
77+
wait "$PID" || true
78+
"$STASH" decrypt backfill --table users --column email_encrypted
79+
80+
REMAINING=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM users WHERE email_encrypted IS NOT NULL AND email IS NULL")
81+
if [ "$REMAINING" != "0" ]; then
82+
echo "FAIL: ${REMAINING} rows still undecrypted" >&2
83+
exit 1
84+
fi
85+
ROUNDTRIP=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg(email, ',' ORDER BY id)) FROM users")
86+
if [ "$ROUNDTRIP" != "$ORIGINAL" ]; then
87+
echo "FAIL: decrypted plaintext does not match the original values" >&2
88+
exit 1
89+
fi
90+
echo "OK: all 5000 rows decrypted back to their original plaintext"
91+
92+
echo "==> 9. Reverse status"
93+
"$STASH" decrypt status
94+
95+
echo "==> 10. Switch application reads to email (manual deploy step)"
96+
97+
echo "==> 11. Generate and apply the encrypted-column drop"
98+
"$STASH" decrypt drop --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle"
99+
DECRYPT_DROP=$(ls -t "$(pwd)/drizzle"/*_drop_encrypted_users_email_encrypted.sql | head -1)
100+
psql "$DATABASE_URL" -f "$DECRYPT_DROP" >/dev/null
101+
GONE=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'users' AND column_name = 'email_encrypted'")
102+
if [ "$GONE" != "0" ]; then
103+
echo "FAIL: email_encrypted still exists" >&2
104+
exit 1
105+
fi
106+
echo "OK: users is plaintext again"
57107

58108
echo "==> Done."

‎languages/typescript/packages/cli/src/bin/main.ts‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,11 @@ Commands:
128128
encrypt backfill Resumably encrypt plaintext into the encrypted column
129129
encrypt drop Generate a migration to drop the plaintext column
130130
131+
decrypt status Show per-column reverse-migration status (phase, progress, undecryptable)
132+
decrypt plan Plan (or generate) the plaintext twins for encrypted columns
133+
decrypt backfill Resumably decrypt ciphertext into the plaintext column
134+
decrypt drop Generate a migration to drop the encrypted column
135+
131136
env Mint deployment credentials and print them as env vars
132137
133138
Options:
@@ -466,6 +471,92 @@ async function runEncryptCommand(
466471
}
467472
}
468473

474+
async function runDecryptCommand(
475+
sub: string | undefined,
476+
flags: Record<string, boolean>,
477+
values: Record<string, string>,
478+
) {
479+
switch (sub) {
480+
case 'status': {
481+
const { decryptStatusCommand } = await requireStack(
482+
() => import('../commands/decrypt/status.js'),
483+
)
484+
await decryptStatusCommand()
485+
break
486+
}
487+
case 'plan': {
488+
const { decryptPlanCommand } = await requireStack(
489+
() => import('../commands/decrypt/plan.js'),
490+
)
491+
await decryptPlanCommand({
492+
table: values.table,
493+
column: values.column,
494+
plaintextColumn: values['plaintext-column'],
495+
migrationsDir: values['migrations-dir'],
496+
dryRun: flags['dry-run'],
497+
})
498+
break
499+
}
500+
case 'backfill': {
501+
const table = requireValue(values, 'table')
502+
const column = requireValue(values, 'column')
503+
const { decryptBackfillCommand } = await requireStack(
504+
() => import('../commands/decrypt/backfill.js'),
505+
)
506+
// Through requireStack: the bound comes from @cipherstash/migrate,
507+
// which imports @cipherstash/stack.
508+
const { InvalidDecryptChunkSizeError, parseDecryptChunkSize } =
509+
await requireStack(
510+
() => import('../commands/decrypt/lib/chunk-size.js'),
511+
)
512+
let chunkSize: number | undefined
513+
if (values['chunk-size'] !== undefined) {
514+
// Parsed here, where the raw text is still in hand, so the error can
515+
// quote exactly what was typed.
516+
try {
517+
chunkSize = parseDecryptChunkSize(values['chunk-size'])
518+
} catch (error) {
519+
if (!(error instanceof InvalidDecryptChunkSizeError)) throw error
520+
p.log.error(error.message)
521+
throw new CliExit(1)
522+
}
523+
}
524+
await decryptBackfillCommand({
525+
table,
526+
column,
527+
plaintextColumn: values['plaintext-column'],
528+
pkColumn: values['pk-column'],
529+
chunkSize,
530+
confirmDualWritesDeployed: flags['confirm-dual-writes-deployed'],
531+
force: flags.force,
532+
confirmPlaintextCurrent: flags['confirm-plaintext-current'],
533+
continueOnUndecryptable: flags['continue-on-undecryptable'],
534+
})
535+
break
536+
}
537+
case 'drop': {
538+
const table = requireValue(values, 'table')
539+
const column = requireValue(values, 'column')
540+
const { decryptDropCommand } = await requireStack(
541+
() => import('../commands/decrypt/drop.js'),
542+
)
543+
await decryptDropCommand({
544+
table,
545+
column,
546+
plaintextColumn: values['plaintext-column'],
547+
migrationsDir: values['migrations-dir'],
548+
uninstallEql: flags['uninstall-eql'],
549+
})
550+
break
551+
}
552+
default:
553+
p.log.error(`${messages.decrypt.unknownSubcommand}: ${sub ?? '(none)'}`)
554+
console.log()
555+
console.log(HELP)
556+
throw new CliExit(1)
557+
}
558+
}
559+
469560
function requireValue(values: Record<string, string>, key: string): string {
470561
const v = values[key]
471562
if (!v) {
@@ -610,6 +701,9 @@ async function dispatch(
610701
case 'encrypt':
611702
await runEncryptCommand(subcommand, flags, values)
612703
break
704+
case 'decrypt':
705+
await runDecryptCommand(subcommand, flags, values)
706+
break
613707
case 'schema':
614708
await runSchemaCommand(subcommand, flags, values)
615709
break

0 commit comments

Comments
 (0)