fix(propagation): bound /txs chunks by payload bytes, not just tx count

[galt-tr]

What Changed

• services/propagation/chunking.go (new): planChunks slices a batch into contiguous spans bounded by both teranode_max_batch_size transactions and the new teranode_max_batch_bytes payload bytes. A single transaction larger than the byte cap still travels, alone in its own chunk, with a Warn carrying its size. broadcastInChunks now calls it, so the dispatcher and the reaper rebroadcast path both inherit the cap.
• config/config.go: new propagation.teranode_max_batch_bytes (default 16 MiB) and the teranode_max_batch_size default lowered 1024 → 1000. Both defaults are exported (DefaultTeranodeMaxBatchSize / DefaultTeranodeMaxBatchBytes) and reused by the propagator's non-positive fallback, so code and shipped config cannot disagree. Comments state Teranode's real, strict rules.
• teranode/client.go: SubmitTransactions returns an error satisfying errors.Is(err, ErrBatchTooLarge) for a 413, or a 400 whose bare body is exactly one of Teranode's three limit early-exits (Invalid request body: too many transactions / too much data / too many submissions). Exact match, not prefix — the same handler answers a bare 400 request context cancelled. The existing errUnexpectedStatusCode wrapping is preserved.
• services/propagation/propagator.go: a shape-rejecting peer votes for nobody. Any transaction left without a vote sets needsNarrowing (reusing the existing narrowChunk halving); a chunk of one requeues carrying the peer's text as its reason. A sibling's 200 stays sticky, so "peer A too-large, peer B 200" settles in one round trip. recordBroadcastOutcomes treats a shape rejection as neutral for the circuit breaker (today a mixed A-400/B-200 charged A the slow-track breaker, and an all-peer 400 reset counters via the unanimous-reject arm). New warns (does not clamp) when a configured cap reaches Teranode's limit, and the startup Info line prints both effective caps.
• parkExhaustedRequeues: the "parent not yet accepted by the network" wording is now reserved for a missing-parent line (mirroring giveUpReason); any other retry reason is quoted verbatim. This was already wrong for STORAGE_ERROR infra lines.
• metrics/metrics.go: arcade_propagation_chunk_bytes histogram (dedicated 4 KiB–64 MiB buckets with resolution across 1–32 MiB), batch_bytes on the endpoint success/failure logs, and arcade_propagation_chunk_total{fallback="size_rejected"}.
• Docs/config: config.example.yaml documents both keys (it previously omitted teranode_max_batch_size entirely); docs/teranode-error-surfacing.md gains a batch-shape paragraph and a response-level table row; metrics/README.md gains a chunking recipe; the smoke harness records body bytes and assertChunkSize checks both caps.

Why It Was Necessary

Fixes #271. broadcastInChunks split a batch by transaction count only. Teranode's /txs handler (every tag since v0.15.0) enforces maxTransactionsPerRequest = 1024 and maxDataPerRequest = 32 MiB, both checked with >= before each read — so the effective rules are strictly fewer than 1024 transactions and strictly under 32 MiB (arcade's shipped default of exactly 1024 was itself refused; production runs 25, so this was latent). On a trip Teranode answers a bare 400 text body after it has already dispatched every transaction it read, and discards the per-tx error list. Arcade could not parse that body, so the whole chunk classified "no per-tx vote → requeue" and was re-sent at the same size until retry_max_attempts parked it at PENDING_RETRY, then the reaper repeated it in 200-tx layers. Because an all-peer bare 400 counted as a unanimous reject, the loop was also invisible to endpoint health.

Decisions: 16 MiB default is half of Teranode's hard ceiling and matches the Kafka producer cap from #330; local caps are inclusive and the margin lives in the defaults; Teranode stays the oracle for per-tx size policy (no locally invented REJECTED reason); the reactive path exists for peers running a lower propagation_httpBodyLimit or a future upstream change.

Testing Performed

• CGO_ENABLED=1 go test ./... — 32 packages ok, 0 failures.
• go build ./..., go vet ./..., go vet -tags=smoke ./tests/smoke/, gofmt -l clean.
• golangci-lint v2.13.2 (run via go run …@v2.13.2, since the installed 2.12.2 binary predates the Go 1.27 target in go.mod) on services/propagation, teranode, config, metrics: 0 issues.
• New tests, all written red-first: TestPlanChunks_Table / _Invariants (pure chunker incl. exact-fit, lone oversize, disabled caps, random contiguity/disjointness), TestProcessBatch_ChunksByBytes, TestProcessBatch_OversizeTxBroadcastAlone, TestNew_BatchCapDefaults, TestNew_WarnsWhenCapsReachTeranodeLimits, TestReapOnce_RebroadcastChunksByBytes, TestSubmitTransactions_413_IsBatchTooLarge / _400BatchLimitBodies_AreBatchTooLarge / _400OtherBodies_AreNotBatchTooLarge, TestNarrowChunk_SizeRejection_AllAcceptedWithinBound (all four rejection shapes, exactly 3 round trips), TestBroadcast_SizeRejection_ChunkOfOne_RequeuesWithReasonNeverRejects, TestBroadcast_SizeRejection_LosesToSiblingAcceptance, TestRecordBroadcastOutcomes_ShapeRejectIsNeutral, TestBatchShapeRejectedLog_HasSizeFields, TestPropagationBatchCapsBind, TestPropagationChunkBytesRegistered. The dormant batchSizes capture in TestProcessBatch_ChunksOversizedBatch is now asserted.
• Not run here: the smoke suite (-tags=smoke, needs podman) and e2e; the smoke package compiles, vets and lints.

Impact / Risk

• Behaviour change only when a chunk exceeds 16 MiB or a peer refuses by shape; typical traffic (~300 B txs) never reaches the byte cap. Big-transaction batches take more sequential rounds; peak concurrency is unchanged.
• Resident request memory per pod is now bounded at roughly 16 MiB × endpoints × max_parallel_chunks × max_concurrent_batches where it was unbounded before.
• Visible wording change: PENDING_RETRY ExtraInfo for non-parent reasons no longer claims a missing parent.
• Consensus metric: chunks every responder refused by shape no longer count as unanimous_reject.
• Deployments that set teranode_max_batch_size: 1024 explicitly now get a startup Warn (the value is not clamped; narrowing absorbs the refusal at the cost of two extra requests per full chunk). Production sets 25 and is unaffected; the deployment configmap should gain teranode_max_batch_bytes explicitly.
• Exact-match brittleness: a Teranode release that rewords a limit body degrades to today's blind requeue; the strings are pinned in client_batch_limit_test.go with a pointer to the upstream file.

Follow-ups (not in this PR)

• Upstream Teranode: the pre-read >= check refuses an exactly-at-limit body and discards the per-tx list on early exit.
• The existing unplaceable narrowing path still narrows unconditionally; the new size path narrows only when a transaction is unvoted. Unifying them changes tested behaviour and was left alone.

Notifications

• @shruggr (issue author)

[Copilot]

🔵 Needs a closer look

It changes core propagation broadcast/narrowing behavior and error classification across multiple paths (dispatcher + reaper + endpoint health accounting), so a final human review is warranted despite the strong test coverage.

Pull request overview

This PR updates the propagation service’s /txs batching so outbound broadcasts are chunked by both transaction count and total payload bytes, aligning Arcade’s behavior with Teranode’s strict request-shape limits and preventing requeue storms caused by oversized batches.

Changes:

• Add byte-aware chunk planning (planChunks) and wire it into both the dispatcher and reaper rebroadcast paths.
• Introduce propagation.teranode_max_batch_bytes (default 16 MiB) and lower the default teranode_max_batch_size (1024 → 1000), with exported defaults used consistently by config and runtime fallback.
• Detect Teranode batch-shape refusals (413, and specific bare 400 bodies) as ErrBatchTooLarge, triggering narrowing/requeue behavior without misclassifying endpoint health or terminalizing transactions.

File summaries

File	Description
tests/smoke/recording_teranode.go	Record request body byte length for smoke assertions.
tests/smoke/harness.go	Explicitly set shipped batch caps for smoke tests.
tests/smoke/chained_txs_test.go	Assert both count and byte caps for recorded /txs batches.
teranode/client.go	Add ErrBatchTooLarge classification for 413 and exact-match 400 limit bodies.
teranode/client_test.go	Extend non-parseable 4xx coverage to include 413.
teranode/client_batch_limit_test.go	New tests pin exact batch-limit bodies and errors.Is(ErrBatchTooLarge) behavior.
services/propagation/propagator.go	Use byte-aware chunking, emit chunk-bytes metrics/log fields, narrow on shape refusals, adjust requeue wording.
services/propagation/propagator_test.go	Assert captured batch body sizes are bounded and sum correctly.
services/propagation/chunking.go	New: planChunks, span fullness logic, and rawTxsBytes helper.
services/propagation/chunking_test.go	New: table + property/invariant tests for byte-aware chunk planning.
services/propagation/batch_shape_rejection_test.go	New: end-to-end tests for narrowing/requeue/health neutrality on shape refusals.
services/propagation/batch_bytes_test.go	New: tests proving chunking-by-bytes and oversize-tx “alone chunk” behavior.
services/propagation/batch_bytes_reaper_test.go	New: reaper rebroadcast path inherits byte chunking.
metrics/README.md	Document queries/recipes for chunking signals.
metrics/metrics.go	Add arcade_propagation_chunk_bytes histogram + clarify chunk_total semantics.
metrics/metrics_test.go	Ensure chunk-bytes histogram is registered/scrapable.
docs/teranode-error-surfacing.md	Document batch-shape rejection behavior and updated broadcast-mode caps.
config/config.go	Add byte cap config + exported defaults; update defaults wiring and comments.
config/config_test.go	Pin defaults and prove env overrides bind for both new/updated keys.
config.example.yaml	Document both propagation chunk cap keys and their rationale.

Review details

• Files reviewed: 20/20 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.