This document lists the CI jobs, how to tell a test bug from a stack bug, and the failures whose causes are known. Consult it before looking at SUT code: most blackbox failures are test or environment problems, not protocol bugs.
.github/workflows/ci.yml runs on every push and pull request to main.
CMake is the only host build; the Makefile is used only by arm-size.
| Job | What it does |
|---|---|
cmake-ipv4-only |
IPv4-only build without TLS (-DSMALLEST_TCP_IPV6=OFF -DSMALLEST_TCP_TLS=OFF) and its tests; then library-only builds with -DSMALLEST_TCP_TCP=OFF and with -DSMALLEST_TCP_UDP=OFF |
cmake-ipv6-only |
IPv6-only build (-DSMALLEST_TCP_IPV4=OFF) and its tests, including the checks that the IPv4-only protocols refuse to compile; then library-only builds without TCP and without UDP |
arm-size |
make arm-size-all: the Cortex-M0 size benchmarks, arm-check-division (§3.10) and arm-check-links (TLS's objects must not need DTLS's record layer, nor DTLS's TLS's) |
board-nucleo-f429zi |
Cross-builds the NUCLEO-F429ZI firmware of the hardware fuzz job — build only |
cmake-linux, cmake-macos |
The default build (dual stack, TLS with Mbed TLS) and ctest; on Linux also sudo ./build/tests/test_rawsock for the raw-socket driver's live tests |
blackbox-linux |
ARP, IPv4, ICMP, UDP, TCP suites against tcp_echo_demo via run_blackbox.sh, once over TAP and once over the raw socket |
blackbox-validate |
The same suites against the Linux kernel as reference SUT (§1) |
blackbox-ipv6, blackbox-dhcp, blackbox-mdns, blackbox-http, blackbox-tls, blackbox-dtls |
The suites that launch their own demo SUTs, each over TAP and the raw socket, with their interop checks (DTLS against wolfSSL, built by tests/blackbox/build_wolfssl.sh and cached); the IPv6 suite also against an IPv6-only build |
fetchcontent |
Builds and runs examples/fetchcontent against the checkout |
traceability |
scripts/trace.py --strict --markdown: every integration test cites the requirements it verifies, every cited ID exists, every test a requirement row names exists and cites that row, and every MUST row is cited by a test or says why none can verify it (§3.12); the summary is the requirements coverage — per requirement document, the MUST rows verified by a black-box test and by any test, those not observable or not implemented, and those left with none (test-plan.md §0) |
coverage |
Builds with -DSMALLEST_TCP_COVERAGE=ON, runs the integration tests (ctest -L integration) and reports, with gcovr, the lines and branches of src/ they reach — what the API reaches. The MAC drivers (src/driver/) and the Mbed TLS backend (src/tls_crypto_mbedtls.c) are left out. The HTML report is the artifact coverage-integration |
release-check |
scripts/release.py check: the version in net_version.h parses, CHANGELOG.md has its ## [Unreleased] section, and a trial stamp of the next version works; CMake and the compiled library report the version (release-process.md) |
fuzz.yml runs the TCP fuzz suite nightly. release.yml runs when this
workflow completes on main: if every job passed and main is still the
tested commit, it commits the next version, tags it and publishes the
release. The full matrix is in
test-plan.md §3.
The core protocol suites (ARP, IPv4, ICMP, UDP, TCP) run in two CI jobs:
| Job | SUT | What a FAIL means |
|---|---|---|
blackbox-linux |
smallest_tcp binary (tcp_echo_demo) |
SUT bug — our code is wrong |
blackbox-validate |
Linux kernel + socat (reference implementation) |
Test bug — the test assertion is wrong |
The DHCPv4, mDNS, HTTP and IPv6 suites are excluded from
blackbox-validate (they drive our demos, not the kernel), and the TLS,
HTTPS and DTLS suites skip themselves there because no SUT binary is given.
Always check blackbox-validate first. If the same test also fails
against the Linux kernel, fix the test — do not touch SUT code. Only once
the test passes against Linux but still fails against our SUT is it safe to
conclude there is a protocol bug in the stack.
For example, test_ipv4_003 expects ICMP Protocol Unreachable for an
unknown protocol on a unicast datagram. Linux sends it, so the test is
right; if only blackbox-linux fails it, look at the default case of
deliver() in src/ipv4.c.
A bug found this way becomes a failing test first — an integration test traced to the requirement it breaks — and then the fix (test-plan.md §0).
CI failure reported
│
▼
Does blackbox-validate also fail for this test?
YES → Fix the test (assertion, timing, operator precedence)
NO → SUT has a bug; continue to step 3
│
▼
Is it ALL tests failing with "ARP timeout"?
YES → SUT didn't start or TAP not up (see §3.1)
NO → Continue to step 4
│
▼
Is it exactly the FIRST test in a suite failing?
YES → Race condition between socket open and stimulus (see §3.2)
NO → Continue to step 5
│
▼
Read the SUT log, arping/tcpdump the interface, isolate the failing exchange
Symptom: Every test in a suite reports ERROR (not FAILED). The ctx
fixture raises RuntimeError: ARP timeout … during setup, before any test body
runs.
Cause: The SUT is not running, or is not attached to the TAP interface.
No process is answering ARP requests on tap0.
Diagnostic checklist:
| Check | Command | Expected output |
|---|---|---|
| SUT process alive? | pgrep -a tcp_echo_demo |
Shows PID |
| SUT log shows TAP open | cat /tmp/sut.log |
[TAP] Opened tap0 (fd=N) |
/dev/net/tun available |
ls -la /dev/net/tun |
crw-rw-rw- … 10, 200 |
tap0 is UP |
ip link show tap0 |
state UP or state UNKNOWN |
| Binary path correct | ls build/demo/tcp_echo_demo |
file exists |
Most frequent cause — wrong binary path:
CMake mirrors the source tree. demo/tcp_echo/main.c → binary at
build/demo/tcp_echo_demo, not build/tcp_echo_demo.
# WRONG — sudo silently exits with "command not found"
sudo ./build/tcp_echo_demo &
# CORRECT
sudo ./build/demo/tcp_echo_demo &After a clean build, always verify:
find build/ -name tcp_echo_demoLXC container issue: /dev/net/tun may not be forwarded into LXC containers.
Use a KVM VM or enable TUN in the Proxmox container config:
lxc.cgroup2.devices.allow = c 10:200 rwm
— or use the raw-socket driver on a veth pair, which needs no TUN
(tests/blackbox/sut_net.sh up raw).
Symptom: test_arp_001 (or the first test of any suite) times out with
empty Scapy results; the next test passes.
Cause: the stimulus went out before the capture socket was open, so the
SUT's reply was missed. On a loaded runner, opening Scapy's AF_PACKET
socket can take longer than any fixed sleep.
How the suites avoid it: helpers.start_sniffer() starts the
AsyncSniffer and returns only once Scapy's started_callback reports its
capture socket open; send the stimulus after it returns. A test that
starts a sniffer itself and sleeps instead reintroduces the race.
Key lesson: If test N always fails and test N+1 always passes, the problem is almost always a capture race, not a protocol bug.
A helper that verifies a reply's checksum must sum the message with the
checksum field in place: a valid one then sums to 0xFFFF (all ones in
one's complement). Zeroing the field before summing tests only whether the
stored checksum is 0 and accepts any reply (_icmp_checksum_ok() in
test_icmp_conform.py):
# WRONG — zeros the checksum, only catches 0x0000
buf = bytearray(raw)
buf[2] = buf[3] = 0
s = sum(buf[i] | buf[i+1] << 8 for i in range(0, len(buf), 2))
return (~s & 0xFFFF) == 0xFFFF
# CORRECT — include the stored checksum in the sum
s = sum(raw[i] | raw[i+1] << 8 for i in range(0, len(raw), 2))
return s == 0xFFFFValidate such helpers against blackbox-validate: Linux generates correct
checksums, and the helper must accept them and reject corrupt ones.
Symptom: TypeError: unsupported operand type at runtime in a Scapy
frame builder, e.g.:
pkt = IP(dst="10.0.0.2") / b"A" + b"B" # TypeErrorCause: Python operator precedence. / (used by Scapy for layer
stacking) binds tighter than + (bytes concatenation). The expression
parses as (IP / b"A") + b"B", where + tries to concatenate a Scapy
packet with a bytes object.
Fix: Parenthesise the bytes payload:
pkt = IP(dst="10.0.0.2") / (b"A" + b"B") # correctAny Scapy expression mixing / and + or - needs explicit parentheses
around the bytes operand.
run_blackbox.sh holds the list of core suites (SUITES: ARP, IPv4, ICMP,
UDP, TCP, and the fuzz tests with --fuzz). blackbox-linux and the
nightly post-fuzz regression in fuzz.yml both run it rather than calling
pytest on single files, so a new core suite added to that list runs
everywhere. A suite that drives its own demo gets its own job instead (§7).
Symptom: test_tcp_014_data_echo_seq_ack, test_tcp_005_graceful_close_active,
and test_tcp_041_out_of_window_segment_gets_ack fail. Tests that only need
a SYN-ACK (e.g. test_tcp_002, test_tcp_076) continue to pass.
A capture on the harness interface (tcpdump -i "$TEST_IF" -n tcp) shows
a RST from 10.0.0.100 that the test never sent, between the SUT's SYN-ACK
and Scapy's ACK.
Cause: 10.0.0.100 (Scapy's source address) is assigned to the
harness interface (sut_net.sh up does ip addr add 10.0.0.100/24).
When the SUT sends its SYN-ACK, the Linux kernel sees a segment for a local
address with no socket on the ephemeral destination port (50001–50N), and
answers with a RST. That RST reaches the SUT before Scapy's handshake
ACK, so the SUT returns to LISTEN, and every later data or FIN segment is
answered with a RST (ACK to LISTEN) instead of an echo.
Tests that pass despite this:
test_tcp_002/076/082— only check the SYN-ACK, which arrives before the RSTtest_tcp_078— vacuously: no payload in the RST replies, so theassert tcp_payload_len <= small_mssloop never runstest_tcp_097— vacuously: no echo data → no ACK sent → no retransmits
What prevents it: sut_net.sh up <driver> --rst-drop drops the
kernel's RSTs leaving through the harness interface only (blackbox-validate
uses the same rule on veth-test):
sudo iptables -A OUTPUT -p tcp --tcp-flags RST RST -o "$TEST_IF" -j DROPScoping it with -o is critical: the SUT's own RSTs (sport=7 →
dport=ephemeral) stay intact, so test_tcp_031 and test_tcp_072 still
see them. sut_net.sh down removes the rule. The HTTP, TLS and DTLS
suites use the host's own TCP stack as the client and run without
--rst-drop.
Diagnostic tip: a RST from 10.0.0.100 that the test did not send, right after the SUT's SYN-ACK, is the kernel's; if the only RSTs are the test's own, look elsewhere.
Symptom: test_dhcpv4_conform.py::test_discover_retransmit fails with:
AssertionError: XID changed across retransmit: 0xcd305e6a → 0x25dbfac1
Cause: the client chose a new transaction ID for a retransmission.
RFC 2131 §4.1 leaves that to the implementation; this client keeps one
xid for an exchange, so an OFFER that answers the first DISCOVER still
matches after a retransmission (the test is sut_specific).
dhcpv4_client.c draws a new xid only when an exchange starts
(start_selecting(), when discovery begins or begins again), never in the
retransmission path of dhcpv4_client_tick().
A field that pairs a reply with a request (XID, sequence number, ISN) is best kept for the life of the exchange; a request/response protocol gets a retransmission test for it.
Symptom: in the raw socket leg, ping/arping and the Scapy suites
pass, but nc, the HTTP suite or the interop scripts time out — everything
where the host kernel sends TCP or UDP to the SUT.
Cause: checksum offload. On a veth pair the kernel leaves the TCP/UDP
checksum partial for the "NIC" to finish; unfinished, the stack drops the
segment. Scapy computes full checksums, so its tests are unaffected.
src/driver/rawsock.c reads each frame with a virtio_net_hdr
(PACKET_VNET_HDR) and finishes the checksum when the kernel flags it
(VIRTIO_NET_HDR_F_NEEDS_CSUM). test_rawsock's
test_live_kernel_{udp,tcp}_checksum_valid catch a regression here
without any blackbox run.
Also: a TAP device serves one process only; a raw socket does not. A
leftover SUT (e.g. the last dhcp_echo_demo) makes the next TAP suite fail
to open tap0 but goes unnoticed on the raw link. conftest.py stops the
DHCP SUT at session end for this reason.
Symptom: small exchanges pass, but a large transfer from the host (the
TLS suite's 40 kB echo or a full 16 kB record) fails in the raw socket
leg: the SUT sends bad_record_mac, or an echo comes back altered.
Cause: the kernel sends bulk data on a veth pair as GSO super-frames,
which the raw-socket driver drops (they exceed one Ethernet frame). The
kernel then retransmits with different segment boundaries, so segments
overlap data already received. TCP must take only the bytes from RCV.NXT
on: data_input() in tcp.c (RFC 9293 §3.10.7.4, step 7) skips the bytes
before RCV.NXT and drops a segment that starts after a gap, and
fin_input() takes a FIN only in sequence. If overlapping bytes are
appended twice, TLS's record MAC catches it where a plain TCP echo would
pass the repeated bytes on silently — look at data_input() first.
test_tcp's in-order delivery tests cover it.
Also on a persistent test host: only one of tap0 / veth-test may
hold 10.0.0.100. A downed tap0 that keeps the address leaves a
linkdown route the kernel still uses, and the raw leg cannot reach the
SUT — sut_net.sh down tap before up raw.
Symptom: make arm-size-all ends with These objects call a library divide (no division on Cortex-M0): and a list of object files.
Cause: a / or % on a run-time value in one of them. Cortex-M0 has
no divide instruction, so the compiler calls a libgcc helper
(__aeabi_uidiv, __aeabi_uidivmod, …) — slow, and several hundred bytes
of flash. arm-check-division runs arm-none-eabi-nm -u on every
benchmark object and on every source in src/ compiled dual stack.
Fix: rewrite the expression without division — shifts and masks for
powers of two, a comparison and subtraction for wrap-around, scaling for
random ranges (net_random_below()), subtraction loops for decimal output
(net_u32_to_dec()). See coding-rules.md §4.
Reproduce locally with rm -rf build/arm && make arm-check-division (make
compares file times to the second, so remove the objects after an edit).
The code, a test or a demo uses something that exists only with
NET_USE_IPV6 (for example net->ip6, the udp6 API, or
http_request_t.remote_ip6) outside #if NET_USE_IPV6. Reproduce with
cmake -S . -B build-v4 -DSMALLEST_TCP_IPV6=OFF -DSMALLEST_TCP_TLS=OFF.
The same job's no-TCP and no-UDP builds catch the equivalent for
NET_USE_TCP and NET_USE_UDP in the libraries
(configuration.md §5).
cmake-ipv6-only is the same for NET_USE_IPV4: reproduce with
-DSMALLEST_TCP_IPV4=OFF.
scripts/trace.py --strict prints a problem: line for each break in the
trace and exits 1:
| Problem | Fix |
|---|---|
An integration test cites no requirement |
Add the REQ IDs the test verifies to the comment above it |
A test cites REQ-…, which no document defines |
Correct the ID, or add the row to its document under docs/requirements/ |
A row names <test>, which is no test |
The Test ID column of the row names a test that was renamed or removed: name the tests that verify the row |
A row names <test>, which does not cite it |
Add the row's ID to that test's comment (for a unit or blackbox test, anywhere in its file) — if the test does verify the row |
A row has a placeholder for a test |
Replace TEST-XXX-NNN in the Test ID column with the names of real tests |
A MUST row is verified by no test and does not say why |
Write a test that cites it; or, if no test can verify it, say so in its Test ID column: — (not observable: <why>) or — (not implemented) |
Reproduce with python3 scripts/trace.py --strict;
python3 scripts/trace.py --untested <doc> lists the MUST rows of one
document that are left with no test.
# List last 5 runs and their conclusions
gh run list --limit 5 --json status,conclusion,databaseId,headSha,displayTitle \
| jq '.[] | {id: .databaseId, conclusion, title: .displayTitle}'
# Show only the failing jobs' logs
gh run view <RUN_ID> --log-failed
# Get the full log for a specific job (e.g. blackbox-linux)
gh run view <RUN_ID> --job blackbox-linux --log | less
# The run_blackbox.sh summary is written to GITHUB_STEP_SUMMARY and
# also captured in /tmp/blackbox.out inside the runner. The CI job
# appends the last 12 KB to the step summary — read it via:
gh run view <RUN_ID> --json jobs \
| jq '.jobs[] | select(.name | contains("Blackbox")) | .steps[] | select(.name | contains("summary")) | .conclusion'Tests decorated with @pytest.mark.sut_specific depend on smallest_tcp's
specific timer values or behaviour and are excluded from
blackbox-validate.
| Test | Why sut_specific |
|---|---|
test_tcp_090_syn_retransmit_on_timeout |
Waits 1.5 s, then up to 5 s, for the SYN-ACK retransmission; SUT initial RTO = 1 s (NET_DEFAULT_TCP_RTO_INIT_MS), Linux starts at 1–3 s |
test_tcp_085_persist_probe_on_zero_window |
Expects a probe within a few seconds; SUT persist interval starts at 1 s, Linux at 5 s |
test_tcp_078_sut_honors_our_mss |
The Linux kernel ignores a tiny peer MSS (100) on veth/loopback with TSO |
test_ipv4_003_unknown_proto_icmp_unreachable, test_ipv4_007_outbound_df_bit_set |
Behaviour of this stack that the kernel reference does not share |
test_http_021_idle_connection_times_out |
Waits for http_demo's 10 s request timeout |
test_dhcpv4_conform.py: test_ack_binds_ip, test_nak_triggers_rediscover, test_wrong_xid_offer_ignored, test_discover_retransmit, test_request_contains_server_id |
Depend on dhcp_echo_demo |
When adding new timer-dependent tests, always ask: "Would this pass against a
standard Linux kernel with default RFC-compliant timer values?" If not, add
@pytest.mark.sut_specific.
Defined in tests/blackbox/conftest.py. Autouse — runs for every test.
[test body runs]
After test: sleep 100 ms ← gives SUT time to reset state / re-enter LISTEN
If failures appear between tests (the SUT still in the previous test's state), lengthen the post-test sleep. Failures of the first test of a session are a capture race instead (§3.2).
When implementing a new protocol and adding blackbox tests:
- ☐ Write
tests/blackbox/test_<proto>_conform.py; each test's docstring or comment names the REQ IDs it verifies (scripts/trace.py) - ☐ If it is a core suite run against
tcp_echo_demo, add it toSUITESintests/blackbox/run_blackbox.sh(§3.5) - ☐ Otherwise add a
blackbox-<proto>job to.github/workflows/ci.yml, with the TAP and raw-socket matrix legs (tests/blackbox/sut_net.sh) - ☐ If the suite can run against the Linux kernel, let
blackbox-validaterun it; if it drives one of our demos, add it to that job's--ignorelist (or make it skip without its SUT option) - ☐ Add a Linux sanity check or interop step (kernel tool, 2–5 s) to the new job
- ☐ Update
docs/test-plan.mdin the same commit — suite table, CI matrix, coverage table - ☐ Update
README.md's blackbox instructions if the suite needs a new option or SUT - ☐ Add
@pytest.mark.sut_specificto any timer-dependent tests - ☐ Verify
blackbox-validatepasses before merging (tests are correct)