feat: cell level das#9160
Conversation
Add PartialDataColumnHeader, PartialDataColumnSidecar, and PartialDataColumnPartsMetadata SSZ container types to the fulu fork definitions per consensus-specs PR #4558.
…-cells feature flags Gate cell-level DAS behind --enable-partial-columns (default: false). Control eager cell publishing with --eagerly-publish-cells (default: true).
Extend BlockInputColumns with: - partialHeader field for validated PartialDataColumnHeader - cellsCache for accumulating individual cells per column - addCells() method that auto-promotes to full DataColumnSidecar - addPartialHeader()/getPartialHeader()/hasPartialHeader() accessors - createFromPartialHeader() static factory
Allow creating/retrieving BlockInputColumns from a PartialDataColumnHeader, enabling the gossip pipeline to track partial columns before any full column or block has been seen.
…e gossip pipeline Add new GossipType that shares the same topic string as data_column_sidecar but routes to its own handler. Wire up gossip queue config and processor routing with bypassQueue for low-latency handling.
Add validation functions per consensus-specs PR #4558: - verifyPartialDataColumnHeaderInclusionProof() - verifyPartialDataColumnSidecarKzgProofs() - validateGossipPartialDataColumnHeader() - full header validation - validateGossipPartialDataColumnCells() - cell validation Add error codes for partial-specific failures.
…th2Gossipsub integration Add gossip handler for GossipType.partial_data_column_sidecar that: - Deserializes PartialDataColumnSidecar - Validates header (proposer sig, inclusion proof, slot checks) - Validates cells (KZG proof verification) - Accumulates cells in BlockInputColumns via addCells() - Emits ChainEvent.publishDataColumns on column completion Integrate partial messages into Eth2Gossipsub: - Listen for gossipsub:partial-message events - Route to partial handler via synthetic pendingGossipsubMessage - Subscribe with partial support on data column topics - Add publishPartialMessage() method Wire enablePartialColumns/eagerlyPublishCells through NetworkConfig.
Add dataColumnToPartialSidecar(), computePartialMessageGroupId(), and buildPartsMetadataBytes() utility functions for constructing partial data column messages from full DataColumnSidecars.
…off trigger Fix critical bugs where BitArray was treated as boolean[]: - Use .bitLen instead of .length for bitmap size - Use .get(i) instead of indexing for bit access - Use .toBoolArray() instead of Array.from() for conversion Add data-availability cutoff trigger in partial handler to emit ChainEvent.incompleteBlockInput when timeout is reached, matching the existing data_column_sidecar handler behavior.
…accumulation Tests cover: - addPartialHeader: add/retrieve, no-op on same, throw on mismatch - createFromPartialHeader: correct type, slot, blockRoot, flags - addCells: accumulation, completion, multi-batch assembly, already-complete column, missing header error
Tests cover serialize/deserialize round trips for: - PartialDataColumnHeader (default and populated) - PartialDataColumnSidecar (empty header, with header and cells) - PartialDataColumnPartsMetadata
The partial header may be the first gossip item for a block, carrying the kzg_commitments needed for getBlobsV3. Trigger it so the EL can provide locally available blobs before relying on cell exchange.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for partial data column dissemination in the beacon node, enabling cell-level Data Availability Sampling (DAS) via gossipsub partial messages. The changes include new SSZ types for partial data columns, validation logic for partial headers and cells, and a new publisher component to manage partial column dissemination. My review identified a few areas for improvement: the loop in BlockInputColumns could be more idiomatic, the header broadcasting logic in gossipHandlers.ts is duplicated and should be refactored, and the peer-to-peer publishing in PartialColumnPublisher could be parallelized for better performance.
| for (let blobIdx = 0; blobIdx < cellsPresentBitmap.length; blobIdx++) { | ||
| if (cellsPresentBitmap[blobIdx]) { | ||
| if (!cellMap.has(blobIdx)) { | ||
| cellMap.set(blobIdx, {cell: cells[cellIdx], proof: proofs[cellIdx]}); | ||
| } | ||
| cellIdx++; | ||
| } | ||
| } |
There was a problem hiding this comment.
The loop iterates over cellsPresentBitmap.length, but cellsPresentBitmap is a boolean[]. While this works, using for...of or forEach is more idiomatic in TypeScript when the index is not strictly required for the logic inside the loop, or if you want to avoid manual index management.
| if (shouldBroadcastHeader && header !== undefined && !hasCells) { | ||
| broadcastedPeers = await partialColumnPublisher.broadcastHeaderAcrossCustodySubnets( | ||
| partialSidecar, | ||
| columnIndex, | ||
| ssz.phase0.BeaconBlockHeader.hashTreeRoot(header.signedBlockHeader.message), | ||
| header.signedBlockHeader.message.slot, | ||
| Array.from( | ||
| new Set( | ||
| chain.custodyConfig.custodyColumns.map( | ||
| (custodyColumn) => custodyColumn % config.DATA_COLUMN_SIDECAR_SUBNET_COUNT | ||
| ) | ||
| ) | ||
| ).sort((a, b) => a - b) | ||
| ); | ||
| } |
There was a problem hiding this comment.
The logic for broadcasting headers is duplicated in two places within this gossip handler. Consider extracting this into a helper function to improve maintainability and reduce the risk of inconsistencies.
| private async publishTrackedPartialOnSubnet( | ||
| subnet: SubnetID, | ||
| blockRoot: Uint8Array, | ||
| blockRootHex: RootHex, | ||
| slot: number, | ||
| skipPeers: ReadonlySet<PeerIdStr>, | ||
| trigger: PartialPublishTrigger | ||
| ): Promise<void> { | ||
| const {topic} = this.getTopicForSubnet(subnet, slot); | ||
| const peers = await this.core.getPartialPeers(topic); | ||
|
|
||
| for (const peerId of peers) { | ||
| if (skipPeers.has(peerId)) { | ||
| continue; | ||
| } | ||
|
|
||
| await this.publishTrackedPartialToPeer(peerId, subnet, blockRoot, blockRootHex, slot, trigger); | ||
| } | ||
| } |
There was a problem hiding this comment.
The publishTrackedPartialOnSubnet method iterates over peers and calls publishTrackedPartialToPeer for each. If the number of peers is large, this could lead to many sequential asynchronous calls. Consider using Promise.all to parallelize these network operations.
1 participant
NOTE: not ready for review. Using local dependency for testing. Need to merge upstream PR first and that will rectify the diff. Its still a very large PR though.
Motivation
Implements cell-level dissemination for PeerDAS columns using gossipsub's Partial Messages Extension (consensus-specs#4558, libp2p/specs#685).
Instead of always exchanging full
DataColumnSidecarmessages, peers can selectively exchange individual cells. This reduces bandwidth and enables faster data availability for nodes that only have a subset of blobs.PartialDataColumnSidecar,PartialDataColumnHeader,PartialDataColumnPartsMetadataBlockInputColumnsextended withcellsCachethat accumulates cells and auto-promotes to full columnsGossipType.partial_data_column_sidecarrouted through existing gossip pipelineChainEvent.publishDataColumnson column completion for backwards-compatible forwardingEth2Gossipsublistens forgossipsub:partial-messageevents and subscribes with partial support on data column topics--enable-partial-columns(default: off)Known TODOs
PartialDataColumnHeadervariant (different fields per EIP-7732)