Skip to content

feat(cdk-mintd): add nutshell database migration subcommand#2016

Open
a1denvalu3 wants to merge 11 commits into
cashubtc:mainfrom
a1denvalu3:feat/nutshell-migration-command
Open

feat(cdk-mintd): add nutshell database migration subcommand#2016
a1denvalu3 wants to merge 11 commits into
cashubtc:mainfrom
a1denvalu3:feat/nutshell-migration-command

Conversation

@a1denvalu3

Copy link
Copy Markdown
Contributor

This PR implements a new subcommand migrate-nutshell for cdk-mintd to automatically migrate a Nutshell mint database (SQLite or Postgres) to the configured CDK mint database.

Key Features:

  1. Direct Parsing to CDK Structs: Avoids custom intermediate structs by directly reading raw database query results and parsing them directly into standard, strongly-typed CDK types (MintKeySetInfo, MintQuote, MeltQuote, BlindSignature, BlindedMessage, Proof).
  2. Chunked constant-memory streaming: Reads nutshell's keysets, mint_quotes, melt_quotes, promises, and proofs tables in segments of 2000 rows. Memory consumption stays flat and negligible even when migrating massive production databases (such as Minibits).
  3. Reuse Existing CDK business logic and CRUD: The data is fully forced through CDK's transaction layers (tx.add_mint_quote, tx.add_proofs, tx.update_proofs_state, etc.), maintaining the integrity of all database validations, state transitions, and historic payment/issuance tracking.
  4. Pre-flight & Version Checks:
    • Aborts if the target database is already populated.
    • Restricts nutshell software versions to <= 0.20.1 (customizable constant).
    • Skips the migration of individual keysets generated under nutshell versions < 0.15.0 (since they use different derivation path logic not supported by CDK).
  5. Warnings Summary: If any pre-0.15 keysets, promises, or proofs are skipped, a clear summary warning is printed at the end of the migration detailing exactly how many records were skipped.
  6. Robust Column parsing: SQLite columns are retrieved as dynamic values to make the parser completely immune to different SQLite column data types (dynamic typing of Integer vs Text in different nutshell versions).

@github-project-automation github-project-automation Bot moved this to Backlog in CDK May 31, 2026
@a1denvalu3 a1denvalu3 force-pushed the feat/nutshell-migration-command branch from 0ad2613 to 7375dbf Compare May 31, 2026 16:06
@codecov

codecov Bot commented May 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1373 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.28%. Comparing base (67047da) to head (0fcc859).
⚠️ Report is 100 commits behind head on main.

Files with missing lines Patch % Lines
crates/cdk-sqlite/src/mint/migrate.rs 0.00% 720 Missing ⚠️
crates/cdk-postgres/src/migrate.rs 0.00% 628 Missing ⚠️
crates/cdk-mintd/src/migrate.rs 0.00% 25 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2016      +/-   ##
==========================================
+ Coverage   69.54%   70.28%   +0.74%     
==========================================
  Files         353      359       +6     
  Lines       69975    75479    +5504     
==========================================
+ Hits        48666    53053    +4387     
- Misses      21309    22426    +1117     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ye0man ye0man moved this from Backlog to Needs Review in CDK Jun 3, 2026
@ye0man ye0man added this to the 0.18.0 milestone Jun 3, 2026
@ye0man ye0man requested a review from crodas June 4, 2026 15:48
@a1denvalu3 a1denvalu3 marked this pull request as draft June 5, 2026 12:20
@a1denvalu3

Copy link
Copy Markdown
Contributor Author

Architectural Note: Standalone Fuzzer Design & Nutshell Mint Lifecycle Spawning

During my review and cleanup of the nutshell-to-CDK database migration integration test (nutshell_migration_fuzzer.rs), I wanted to highlight the current design approach and present an alternative for future consideration.

Path A: Programmatic, Self-Contained Spawning in Rust (Current Design)

Currently, the fuzzer test manages the entire lifecycle of both the Nutshell mint (via Python Poetry or Docker container) and the subsequent migrated CDK mint directly within the Rust process (#[tokio::test]).

  • How it works:
    1. Spawns Nutshell (Docker or Python module) in the background.
    2. Funds and fuzzes the active Nutshell mint with mock wallets.
    3. Shuts down Nutshell cleanly.
    4. Runs the database migration directly against the target SQLite/Postgres.
    5. Boots up a CDK Axum mint pointing to the migrated DB on the same port in-process.
    6. Fuzzes again with the same wallets to verify exact balances and spendability.
  • Pros: Highly self-contained and portable. Any developer or IDE can execute cargo test -p cdk-integration-tests --test nutshell_migration_fuzzer without needing any external bash scripting or orchestrators.

Path B: Bash/Shell Orchestration & Multi-Phase Binaries (Alternative Design)

An alternative would be to strip all setup, process management, and docker lifecycle handling out of the Rust codebase, delegating it to an external bash script (similar to the way nutshell_wallet_itest.sh coordinates tests).

  • How it would work:
    1. Bash script starts the Nutshell container.
    2. Cargo test runs "Phase 1" fuzzing (wallet funding and operations).
    3. Bash script stops the Nutshell container.
    4. Bash script executes cdk-mintd migrate-nutshell ... CLI command to run the migration.
    5. Bash script starts cdk-mintd daemon pointing to the migrated DB.
    6. Cargo test runs "Phase 2" verification (wallet spendability).
    7. Bash script terminates everything and cleans up.
  • Pros: Less custom process management / subprocess spawning logic in Rust.
  • Cons: Harder to run locally (developers cannot just run cargo test), and more fragile under CI/CD because a failed shell command or early exit might easily orphan background processes or leave ports bound.

Both options are viable, but the current Path A keeps integration testing robust, fully portable, and extremely simple to run with a single Cargo invocation.

@a1denvalu3 a1denvalu3 force-pushed the feat/nutshell-migration-command branch 6 times, most recently from a34344b to 12754fc Compare June 6, 2026 12:59
@a1denvalu3 a1denvalu3 force-pushed the feat/nutshell-migration-command branch from 12754fc to 653c105 Compare June 6, 2026 14:01
@a1denvalu3 a1denvalu3 marked this pull request as ready for review June 8, 2026 09:37
@prusnak

prusnak commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

This is amazing! 🫶

Comment thread .github/workflows/nutshell_itest.yml Outdated

@cdk-bot cdk-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified findings approved for disclosure:

  • Onchain melt quote migration drops rows with empty fee_options (high) - Onchain melt quotes in a Nutshell database are not migrated, causing systematic loss of those quote records during the new migration flow.
  • Mint quote migration doubles paid and issued totals (high) - Migrated paid/issued mint quotes record twice the actual paid and issued amount, corrupting quote accounting and quote state after migration.

Comment thread crates/cdk-sqlite/src/mint/migrate.rs
Comment thread crates/cdk-postgres/src/migrate.rs
// Start main database transaction after keysets are committed to avoid SQLite lock deadlock
let mut tx = MintDatabase::begin_transaction(&db).await?;

// 4. Chunked Migration of Mint Quotes

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paid/issued mint quotes are migrated with doubled totals. The reader builds quote_obj with amount_paid/amount_issued already set to the legacy amount, but after add_mint_quote the migration calls add_payment(amount, ...) and add_issuance(amount). Those mutators are additive on the in-memory quote, so a paid quote goes from amount to 2 * amount before update_mint_quote writes quote.amount_paid()/quote.amount_issued() back to the row. The initial insert does not persist the pre-populated totals, so the fix is to either construct the quote with zero counters and replay the payment/issuance events, or persist the legacy totals directly without replaying additive events.

let mut skipped_promises_count = 0;
let mut skipped_proofs_count = 0;
let seen_paid_pending_lookup_ids = RefCell::new(HashSet::new());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paid/issued mint quotes are migrated with doubled totals. The reader builds quote_obj with amount_paid/amount_issued already set to the legacy amount, but after add_mint_quote the migration calls add_payment(amount, ...) and add_issuance(amount). Those mutators are additive on the in-memory quote, so a paid quote goes from amount to 2 * amount before update_mint_quote writes quote.amount_paid()/quote.amount_issued() back to the row. The initial insert does not persist the pre-populated totals, so the fix is to either construct the quote with zero counters and replay the payment/issuance events, or persist the legacy totals directly without replaying additive events.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs Review

Development

Successfully merging this pull request may close these issues.

5 participants