Skip to content

UPDATING-RETH.md

Latest commit

 

History

History
323 lines (264 loc) · 16.1 KB

UPDATING-RETH.md

File metadata and controls

323 lines (264 loc) · 16.1 KB
Raw

Updating the reth dependency

The Rust workspace pins ~70 reth-* crates to a single git ref, from upstream paradigmxyz/reth or from an OP-maintained fork of it, in rust/Cargo.toml — tag = "vX.Y.Z" when tracking a release (the normal case; self-documenting), or rev = "<sha>" when pinning a non-release commit. This guide describes how to bump that pin safely and keep the shared revm/alloy versions in sync with it.

When to update

  • Picking up a fix or feature merged upstream that op-reth depends on.
  • Periodic catch-up to limit drift before it becomes painful.
  • After your own reth-side change has merged upstream.

Prefer the latest upstream release tag when one exists that contains the changes you need. Tags are stable artifacts with corresponding upstream releases and the most predictable downstream behavior. Fall back to a merge commit on main only when no tagged release covers the needed change yet (e.g. urgently picking up a fix). Avoid pinning to an unmerged PR branch for anything we want to land — the merge commit on main is at least the version main's CI actually validated.

Procedure

  1. Pick the target revision. Honor an explicit release tag, commit SHA, or merged upstream PR first. Otherwise prefer the latest suitable upstream release. For a PR, use its merge commit; use current main only when no release contains the needed change and no specific stable target was given.

  2. Audit what the current pin carries before moving off it. If the current pin is not an ancestor of the target (e.g. it was an unmerged-PR commit), find what it carried — the monorepo commit that introduced it (git log -S '<pin>' -- rust/Cargo.toml) and the upstream PR (gh api repos/paradigmxyz/reth/commits/<sha>/pulls) — and verify each carried change is contained in, or explicitly superseded by, the target. Never silently drop a fix the pin was carrying.

    The same audit applies to the fork itself. In a checkout of the reth fork (not the monorepo), fetch its upstream remote, find the pinned commit's upstream base, and list its cherry-picks:

    git fetch <upstream-remote> main
OLD_PIN=<old-pin>
OLD_BASE=$(git merge-base "$OLD_PIN" <upstream-remote>/main)
git log --reverse --oneline "$OLD_BASE..$OLD_PIN"

    For each commit, decide whether the selected target contains it or an equivalent. Drop patches already present there; preserve every other patch. If a rewritten or squashed equivalent makes that unclear, ask rather than guess.

  3. Update the pin in rust/Cargo.toml.

    If no fork cherry-picks survive, prefer the upstream release tag. For an existing upstream tag pin, replace the tag. For a fork rev pin, replace both the source and ref:

    cd rust
perl -pi -e 's/tag = "<OLD_TAG>"/tag = "<NEW_TAG>"/g' Cargo.toml
perl -pi -e 's#git = "https://github.com/<OLD_OWNER>/reth", rev = "<OLD_REV>"#git = "https://github.com/paradigmxyz/reth", tag = "<NEW_TAG>"#g' Cargo.toml
grep -nE '<OLD_TAG>|<OLD_REV>|github.com/<OLD_OWNER>/reth' Cargo.toml  # no matches

    If patches must survive, replay and publish them in the reth fork checkout before changing the monorepo pin:

    git switch -c <fork-branch> <new-upstream-rev>
git cherry-pick <retained-sha>...  # oldest first
git push <fork-remote> HEAD:<fork-branch>
git rev-parse HEAD                 # use this as the monorepo rev pin

    Pin that commit with rev = "<sha>". Verify every reth workspace dependency uses the same repository and ref; the mirror checker rejects split pins.

    Use ethereum-optimism/reth's optimism branch for maintenance backports. Rebuild it from the selected upstream release, replay the required runtime patches, then replay the fork's CircleCI/Actions support as one final commit. Keep those CI changes squashed so subsequent rebuilds need only one CI cherry-pick. Preserve the original branch tip locally and use an explicit force-with-lease when publishing a rebuilt branch. Update the maintenance-push compact-codec comparison base in the fork's .github/workflows/compact.yml to the selected release too; comparing against main can test unsupported future formats. Check the fork's own lockfile advisories separately: downstream lockfile fixes do not repair the fork's CI.

    Merging a fix into upstream main does not establish that a patch release contains it. Check the selected tag's ancestry or patch equivalence before removing any backport; a newer tag can still omit the required fix.

    The lockfiles record the resolved commit, so builds remain reproducible.

  4. Sync shared dependency versions to the new rev's pins. reth and the OP Stack share the revm/revm-*, alloy-* (core and main), alloy-eip7928, and the published reth-core crate families (reth-primitives-traits, reth-codecs, reth-rpc-traits, reth-zstd-compressors). reth pins them in its own workspace Cargo.toml; ours must declare the same versions so we build against the same types reth's APIs expose. Read reth's pins at the new rev and update the matching lines in rust/Cargo.toml:

    # from a reth checkout at the new rev:
git show <NEW_REV>:Cargo.toml | grep -E '^(revm|alloy-|reth-)'

    Bump only these crates.io ecosystem crates. Leave the OP-internal path crates (op-revm, op-alloy*, alloy-op-evm, alloy-op-hardforks) alone — they live in-tree under rust/, not on crates.io.

    A version mismatch in the reth-core crates shows up as baffling expected SealedHeader, found a different SealedHeader errors with a "multiple different versions of crate reth_primitives_traits" note. Verify a single version survives unification:

    mise exec -- cargo tree -i reth-primitives-traits@<OLD_MINOR>  # should not match

    Easy to miss: cargo may have already floated these up in Cargo.lock when you bumped the rev — our declared versions are caret ranges ("2.0.4" admits 2.0.5) and reth's higher pin wins unification. So the lock can be correct while the declared versions silently lag, and this step's Cargo.lock diff is often empty. Update the declared versions anyway: it keeps the manifest honest about what we actually build against and signals the sync to downstream consumers (e.g. Hardhat tracking op-revm).

    Conversely, do not bump the version of our own published crates (op-revm, alloy-op-evm, alloy-op-hardforks, op-alloy*) as a side effect of a pin bump. Downstream consumers pin them by version requirement and redirect the source with [patch.crates-io] — Flashbots' op-rbuilder does this for eight of them, against a monorepo tag. If our version stops satisfying their requirement, cargo only warns that the patch went unused and silently resolves the crates.io version instead. Note the 0.x crates are the fragile ones: alloy-op-evm at 0.32.x breaks on a minor bump, while op-revm at 20.x only breaks on a major. If a bump genuinely needs one, call it out in the PR description so the downstream pin can move with it.

    When a bump is genuinely needed, version in-tree crate families as a group. Upstream reth versions every crate from one workspace version; two in-tree families follow the same rule: the published op-reth crates (op-reth, reth-optimism-*, reth-op) and the op-alloy* crates. When any crate of a family needs a new version, bump the whole family to the same version. A lone bump leaves siblings whose API also changed at a version that downstream requirements still accept.

  5. Refresh both lockfiles — the main workspace and the SP1 guest programs workspace each have their own. cargo update -p reth does not work — there is no top-level crate literally named reth in the dependency graph. Pass any real reth subcrate in the main workspace, then refresh revm in the SP1 guest workspace:

    mise exec -- cargo update reth-chainspec
(cd kona/sp1/programs && mise exec -- cargo update revm)

    A targeted cargo update can also advance unrelated dependencies that track a branch. Inspect every changed git source in the lockfile delta. Restore unrelated drift or replace its branch with an intentionally reviewed exact revision before proceeding.

  6. Revisit the upstream slot-preimage seeding API. op-reth/crates/cli/src/commands/slot_preimages_seed.rs uses reth's public SlotPreimages helper directly. Review changes to that helper and its MDBX layout between the old and new reth revisions:

    git -C <reth-checkout> diff <OLD_REV> <NEW_REV> -- \
  crates/stages/stages/src/stages/execution/slot_preimages.rs

    Confirm the import and open/insert_preimages calls remain compatible. Do not reintroduce a local copy of the MDBX layout.

  7. Compile and adapt:

    mise exec -- cargo check --workspace --tests

    Fix each compile error, then re-run. Don't try to predict the full set of breakages in advance — let the compiler walk the dep graph. Each pass surfaces the next crate's errors. Remember the in-tree forks of upstream crates (op-revm, alloy-op-evm, kona's fpvm_evm) — when upstream reworks an API, the right fix is usually to mirror what the new upstream default/eth implementation does, found in the new rev's sources or the cargo registry sources under ~/.cargo/registry/src/.

    Gas accounting and other consensus-adjacent overrides deserve extra rigor: derive the change as (old upstream default → new upstream default) applied onto our override, leaving the OP-specific branches byte-identical — never invent logic. Any new semantic branch (even fork-gated and inert today) gets a unit test, verified red against a deliberately broken variant and green against the real code.

    The compiler will not point you at the overrides that should have changed but didn't. Two things to do by hand:

    • From rust/, just mirrors stale lists mirrors that still record an older pin. Work every entry and advance a tag only after checking its named upstream symbol. See docs/ai/reth-upstream-mirrors.md.
    • For anything touching validation or gas, ask which upstream precondition makes the change safe and whether the OP Stack holds it. Deposits skip validate_env outright, which exempts them from a long list of checks upstream may treat as guaranteed. docs/ai/reth-update-review.md, "The precondition question", has the list and the procedure.

  8. Build, format, and test before pushing:

    mise exec -- cargo build -p op-reth
just fmt-fix && just lint
just test   # unit + doc tests — CI runs both, and doc examples call reth APIs too

    Only the just recipes use the repo-pinned nightly rustfmt — a bare cargo fmt on the stable toolchain reformats unrelated files and fails CI's rust-fmt gate. Also run the test suites of any vendored-workspace crate whose source you touched.

  9. Regenerate the CLI surface snapshot. Most reth bumps add, remove, or re-default upstream CLI flags, and op-reth inherits them silently. The snapshot test rust/op-reth/crates/cli/tests/cli_snapshot.rs fails on any such change so it gets reviewed instead of shipped unnoticed. If the diff is intentional, regenerate and commit the snapshot:

    UPDATE_SNAPSHOT=1 cargo nextest run -p reth-optimism-cli --all-features cli_surface_snapshot

    Review the snapshot diff like code: it is the operator-facing surface of the node. Upstream flags op-reth deliberately rejects (the DENIED_ARGS deny-list in op-reth/crates/cli/src/lib.rs, e.g. --minimal) must stay rejected — they render with a [hidden] marker in the snapshot.

    Report removed flags and changed defaults in the PR's migration notes. The published CLI reference intentionally documents a finalized release, not develop. Do not regenerate it from an unreleased dependency bump; update it through the release generator after the next finalized tag.

Expect upstream churn beyond your target change

A rev bump is rarely "just a rev change." Upstream reth iterates trait signatures, struct fields, re-exports, and feature flags between commits, and any of them can require op-reth-side adaptation. Recent examples observed in the wild:

  • Trait methods gaining new parameters (e.g. FullConsensus::validate_block_post_execution gained block_access_list_hash: Option<B256>; PayloadTypes::block_to_payload gained bal: Option<Bytes>).
  • Associated-type bounds (e.g. PayloadTypes::ExecutionData gaining a From<Self::BuiltPayload> bound, requiring a new From impl).
  • Provider trait additions (e.g. BalProvider becoming a required bound on the engine API's Provider).
  • Struct destructuring (e.g. BlockBuilderOutcome gaining a block_access_list field; BuiltPayloadExecutedBlock.hashed_state / trie_updates losing their Either<Arc<...>, _> wrappers).
  • Renames (e.g. ComputedTrieData::without_trie_input → ::new).
  • Removed re-exports — sometimes the trait itself is deleted upstream while op-reth (and the wider OP Stack) still relies on it. In that case the smallest correct fix is to vendor the trait locally in the consuming crate (with a comment pointing at the upstream PR that removed it). See rust/op-reth/crates/rpc/src/witness.rs for an example.
  • Type changes in test fixtures (e.g. struct fields flipping from u16 to Option<u16>).

None of these are deep redesigns — most fixes are one to three lines — but they need to be done before CI will pass.

Picking the right target commit

In order of preference:

  1. Latest upstream release tag that contains the change you need. Tags correspond to actual upstream releases and have the most predictable downstream behavior. List them with git -C <reth-checkout> tag --sort=-v:refname | head (or gh release list --repo paradigmxyz/reth).

  2. Merge commit on main of a specific PR — for cases where the needed change has been merged but isn't in a release yet (e.g. urgent fix). Find it via gh pr view <N> --repo paradigmxyz/reth --json mergeCommit. The merge commit is stable (PR rebases no longer affect it) and is the version main's CI actually validated.

  3. Current main HEAD — for periodic catch-up only when no suitable release exists.

Avoid pinning to an unmerged PR branch tip for anything we want to land. It moves under us when the PR rebases, may sit on a much newer main than our current pin, and isn't a version any upstream CI has signed off on as a release artifact.

If the only upstream change you need is a single PR but the PR branch is based on much newer main than our pin, consider asking the upstream author to rebase onto an older base, or accept the broader catch-up work as part of the bump.

See also

  • docs/ai/reth-update-review.md — the review guide for these bumps: the risk taxonomy, the deliberately-broken upstream preconditions to check against, and the reth-update-reviewer agent that surfaces upstream changes which should have forced an op- change but didn't.
  • docs/ai/reth-upstream-mirrors.md — the UPSTREAM-MIRROR tags marking OP code that reproduces upstream logic instead of calling it, each recording the upstream version it was last verified against. just mirrors stale turns them into the worklist for a bump.
  • rust/scripts/upstream-mirrors.py — the tag checker behind just mirrors and just check-upstream-mirrors (the latter runs as part of just lint).
  • docs/ai/rust-dev.md — broader Rust workflow (build, test, lint).
  • rust/Cargo.toml — where the pin lives (~70 occurrences).
  • rust/kona/sp1/programs/Cargo.lock — the second lockfile; refresh it when shared revm or alloy anchors move.
  • rust/op-reth/crates/rpc/src/witness.rs — example of vendoring a trait that upstream removed.
  • rust/op-reth/crates/cli/src/commands/slot_preimages_seed.rs — caller of reth's public SlotPreimages seeding API; review the upstream helper on every bump.