feat: adopt official go-whatsonchain SDK in fetch-block-fixture - #344
Merged
Conversation
Replace the tool's bespoke net/http calls to WhatsOnChain with the official github.com/mrz1836/go-whatsonchain v1.3.0 client: - fetchBlockTxIDs uses GetBlockByHash + GetBlockPages (inline first page plus numbered pages), guarded by the existing txCount/coinbase asserts. - Picked txs and their input parents are bulk-fetched via BulkRawTransactionDataProcessor (batches of 20, paced by the client rate limit); the manual 300ms sleeps and 429 retry loop are gone since the SDK handles retry/backoff and rate limiting. - --woc is wired through the SDK's new WithBaseURL option, so it can still target a mirror, proxy, or test server. httpGetBytes remains for the non-WOC teranode datahub fetch. - enrichToEF is now a pure function (parent output injection only). Adds fetch_test.go with hermetic httptest coverage for pagination, bulk batching, EF enrichment and error paths, dedup/determinism, 429 retry, base URL wiring, and run() against the committed block binary. Validated against live WhatsOnChain (bulk /txs/hex works on the free tier; the 1911-tx block returns 100 inline + 1 page).
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No unresolved review comments remain, and the supplied validation supports approval.
Pull request overview
Migrates the fixture generator to the official WhatsOnChain SDK with comprehensive hermetic test coverage.
Changes:
- Replaces bespoke HTTP, pagination, retry, and batching logic.
- Adds tests for fetching, batching, retries, enrichment, and validation.
- Updates dependencies and gitleaks configuration.
File summaries
| File | Description |
|---|---|
tools/fetch-block-fixture/fetch.go |
SDK-based block and transaction pipeline |
tools/fetch-block-fixture/fetch_test.go |
Hermetic tests for the new workflow |
go.mod |
Dependency additions and updates |
go.sum |
Dependency checksum updates |
.gitleaksignore |
Fixture-only false-positive exclusion |
Review details
- Files reviewed: 4/5 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the
fetch-block-fixturedev tool from bespokenet/httpcalls to WhatsOnChain over to the officialgithub.com/mrz1836/go-whatsonchainclient (v1.3.0), and adds hermetic test coverage for the previously-untested WhatsOnChain code path.WhatsOnChain was used in exactly one place in the repo — this fixture generator — and on no runtime/service path. Nothing user-facing or in production changes.
Motivation
merkle_test.goexisted).What changed
fetchBlockTxIDs→GetBlockByHash+GetBlockPages(inline first page plus numbered pages), still guarded by the existingtxCount/ coinbase assertions.BulkRawTransactionDataProcessor(batches of 20, paced by the client's rate limit). The manual sleeps and 429 loop are gone — the SDK handles retry/backoff and rate limiting.--wocis wired through the SDK'sWithBaseURLoption, so it can still target a mirror, proxy, or local test server.httpGetBytesis kept, but now only for the non-WhatsOnChain teranode datahub fetch.enrichToEFis now a pure function (parent-output injection only; no network).Upstream SDK change
WithBaseURLdid not exist in the SDK (the base URL was a hardcoded const). It was added upstream in go-whatsonchain#202 and released asv1.3.0. It is additive and backward-compatible. Adding the dependency pulls in zero new transitive modules (onlytestify+yaml/v3, already in the graph).Testing
fetch_test.go— 15 hermetichttptest-backed tests: pagination, bulk batching (>20), missing/bad-hex handling, EF enrichment + error paths, parent dedup/determinism, 429 retry, base-URL wiring, andrun()driven against the committed block binary.newWOCClient100%,enrichToEF92%,bulkRawTxs88%,buildPickedEFTxs84%,fetchBlockTxIDs83%.POST /txs/hexworks on the free tier without a key; the real 1911-tx block returns 100 txids inline + 1 page — exactly the model the new code reproduces.magex format:fix,go-pre-commit,magex lint(0 issues), andmagex test:race(full suite) all pass.Note on branch contents
Alongside the SDK adoption, this branch also carries two housekeeping commits: a
.gitleaksignoreentry and a repo-widego-sdk v1.4.1 → v1.5.1/go-teranode-p2p-client v0.3.0bump. The full race suite passes with those included.Verification