Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/running_tests/consume/simulators.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,7 @@ uv run consume <engine|rlp> [OPTIONS]
- Help [setting up](../hive/index.md) and [starting Hive in dev mode](../hive/dev_mode.md).
- For an explanation of how the `consume` simulators work, see the [Engine](../running.md#engine) and [RLP](../running.md#rlp) sections in [Running Tests](../running.md).
- Help for relevant options can be found in [Consume Cache and Fixture Inputs](./cache.md) and [Useful Pytest Options](../useful_pytest_options.md).

## Related: Block Building

A separate hive simulator [`build-block`](../running.md#block-building) is also fixture-driven but tests the client's **producer-side** path via the `testing_buildBlockV1` engine-API testing-namespace endpoint, rather than the consumer-side import path that the simulators above exercise.
23 changes: 23 additions & 0 deletions docs/running_tests/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Both `consume` and `execute` provide sub-commands which correspond to different
| [`consume enginex`](#enginex) | Client imports blocks via Engine API in Hive, optimized by client reuse | EVM, block processing, Engine API | Staging, Hive | System test |
| [`consume sync`](#sync) | Client syncs from another client using Engine API in Hive | EVM, block processing, Engine API, P2P sync | Staging, Hive | System test |
| [`consume rlp`](#rlp) | Client imports RLP-encoded blocks upon start-up in Hive | EVM, block processing, RLP import (sync\*) | Staging, Hive | System test |
| [`build-block`](#block-building) | Client builds blocks via Engine API testing namespace in Hive, validated against fixture | EVM, block production, Engine API testing namespace | Staging, Hive | System test |
| [`execute hive`](./execute/hive.md) | Tests executed against a client via JSON RPC `eth_sendRawTransaction` in Hive | EVM, JSON RPC, mempool | Staging, Hive | System test |
| [`execute remote`](./execute/remote.md) | Tests executed against a client via JSON RPC `eth_sendRawTransaction` on a live network | EVM, JSON RPC, mempool, EL-EL/EL-CL interaction (indirectly) | Production | System Test |

Expand Down Expand Up @@ -144,6 +145,28 @@ The `consume sync` command:
5. **Monitors sync progress** and validates that the sync client reaches the same state.
6. **Verifies final state** matches between both clients.

## Block Building

| Nomenclature | |
| -------------- | ------------------------ |
| Command | `build-block` |
| Simulator | `eels/build-block` |
| Fixture format | `blockchain_test_engine` |

The block-building method tests the **producer-side** of an execution client: rather than asking the client to validate and import a pre-built block, it asks the client to build a block from inputs (parent, payload attributes, transactions) and then validates the resulting block field-by-field against the fixture's expected block. This exercises tx ordering, gas accounting, payload assembly, and (for fork ≥ Prague) `executionRequests` derivation.

The endpoint used is `testing_buildBlockV1`, an engine-API testing-namespace method exposed by `ethpandaops/<client>:master` builds (and similar performance forks). It is not part of the standard Engine API — the testing namespace is opt-in and intended for fixture-driven block-building verification.

The `build-block` command, for each valid payload in the fixture:

1. **Builds the block** via `testing_buildBlockV1(parent_hash, payload_attributes, transactions, extra_data)`. The client returns its own constructed `ExecutionPayload`.
2. **Validates execution-dependent fields** of the built payload against the fixture's expected payload (everything except `gas_limit` and `block_hash`, which depend on client-side EIP-1559 elasticity and are validated via a range check separately).
3. **Validates `executionRequests`** for fork ≥ Prague (`engine_newPayloadV4+`).
4. **Imports the fixture block** (not the client-built one) via `engine_newPayloadVX` so the chain advances with the fixture's expected gas limit and block hash.
5. **Advances the chain** via `engine_forkchoiceUpdatedVX`.

This complements `consume engine`: where `consume engine` tests the client's payload-validation path, `build-block` tests its payload-production path against the same fixtures.

## Engine vs RLP Simulator

The RLP Simulator (`eels/consume-rlp`) and the Engine Simulator (`eels/consume-engine`) should be seen as complimentary to one another. Although they execute the same underlying EVM test cases, the block validation logic is executed via different client code paths (using different [fixture formats](./test_formats/index.md)). Therefore, ideally, **both simulators should be executed for full coverage**.
Expand Down
1 change: 1 addition & 0 deletions packages/testing/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ checkfixtures = "execution_testing.cli.check_fixtures:check_fixtures"
check_eip_versions = "execution_testing.cli.pytest_commands.check_eip_versions:check_eip_versions"
consume = "execution_testing.cli.pytest_commands.consume:consume"
protec = "execution_testing.cli.pytest_commands.consume:consume"
build-block = "execution_testing.cli.pytest_commands.build_block:build_block"
checklist = "execution_testing.cli.pytest_commands.checklist:checklist"
generate_checklist_stubs = "execution_testing.cli.generate_checklist_stubs:generate_checklist_stubs"
genindex = "execution_testing.cli.gen_index:generate_fixtures_index_cli"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
"""CLI entry point for the `build-block` pytest-based command."""

from pathlib import Path
from typing import Any, List

import click

from .base import PytestCommand, common_pytest_options
from .processors import (
ConsumeCommandProcessor,
HelpFlagsProcessor,
HiveEnvironmentProcessor,
)


def create_build_block_command() -> PytestCommand:
"""Initialize the build-block command with paths and processors."""
base_path = Path("cli/pytest_commands/plugins/consume")
command_logic_test_paths = [
base_path / "simulators" / "simulator_logic" / "test_via_build.py"
]
return PytestCommand(
config_file="pytest-consume.ini",
argument_processors=[
HelpFlagsProcessor("consume"),
HiveEnvironmentProcessor(command_name="build_block"),
ConsumeCommandProcessor(is_hive=True),
],
command_logic_test_paths=command_logic_test_paths,
)


@click.command(
name="build-block",
Comment on lines +33 to +34
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command does use the testing_buildBlockV1 endpoint, but I think it's rather close to a consume simulator (it still consumes a fixture) so I wondered if it should be a consume sub-command?

I.e., something like uv run consume build-block.

Then the simulator in hive would be eels/consume-build-block.

Imo, when we add a command to fill tests using the testing_buildBlockV1 in the future, that command should be the new build or build-block command. Wdyt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I thought about this a bit but it seems like the opposite of consume imo. It receives a request to build a block and then is validated against what's in the fixture, rather than sending the payloads as they are (consuming) and then validating against some post state after.

I'm not one to argue semantics tbh, all I care is that the functionality is there. I'd love to get ideas from the team and settle on something but the reason I went with a new top-level command is that this feels entirely different to me. I can see the argument since they share a lot of things though. Happy to go either way here.

context_settings={"ignore_unknown_options": True},
)
@common_pytest_options
def build_block(pytest_args: List[str], **kwargs: Any) -> None:
"""Test block building via testing_buildBlockV1."""
del kwargs

cmd = create_build_block_command()
cmd.execute(list(pytest_args))
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ def check_live_port(test_suite_name: str) -> Literal[8545, 8551]:
"eels/consume-engine",
"eels/consume-enginex",
"eels/consume-sync",
"eels/build-block",
}:
return 8551
raise ValueError(
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Pytest configuration for the build-block simulator."""
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
"""
Pytest fixtures for the `build-block` simulator.

Configures the hive back-end & EL clients for block building correctness
testing via the ``testing_buildBlockV1`` endpoint.
"""

import io
from typing import Mapping

import pytest
from hive.client import Client

from execution_testing.fixtures import BlockchainEngineFixture
from execution_testing.fixtures.blockchain import FixtureHeader
from execution_testing.rpc import TestingRPC

pytest_plugins = (
"execution_testing.cli.pytest_commands.plugins.pytest_hive.pytest_hive",
"execution_testing.cli.pytest_commands.plugins.consume.simulators.base",
"execution_testing.cli.pytest_commands.plugins.consume.simulators.single_test_client",
"execution_testing.cli.pytest_commands.plugins.consume.simulators.test_case_description",
"execution_testing.cli.pytest_commands.plugins.consume.simulators.timing_data",
"execution_testing.cli.pytest_commands.plugins.consume.simulators.exceptions",
"execution_testing.cli.pytest_commands.plugins.consume.simulators.engine_api",
)


def pytest_configure(config: pytest.Config) -> None:
"""Set the supported fixture formats for the build-block simulator."""
config.supported_fixture_formats = [BlockchainEngineFixture] # type: ignore[attr-defined]


@pytest.fixture(scope="module")
def test_suite_name() -> str:
"""The name of the hive test suite used in this simulator."""
return "eels/build-block"


@pytest.fixture(scope="module")
def test_suite_description() -> str:
"""The description of the hive test suite used in this simulator."""
return (
"Test block building correctness via the "
"testing_buildBlockV1 endpoint."
)


@pytest.fixture(scope="function")
def client_files(
buffered_genesis: io.BufferedReader,
) -> Mapping[str, io.BufferedReader]:
"""Define the files that hive will start the client with."""
return {"/genesis.json": buffered_genesis}


@pytest.fixture(scope="function")
def genesis_header(fixture: BlockchainEngineFixture) -> FixtureHeader:
"""Provide the genesis header from the fixture."""
return fixture.genesis


@pytest.fixture(scope="function")
def testing_rpc(client: Client) -> TestingRPC:
"""Initialize Testing RPC client for the execution client under test."""
return TestingRPC(f"http://{client.ip}:8545")
Loading
Loading