Skip to content

connection: keep sends out while try_recv reads nonblocking - #575

Open
Mattbusel wants to merge 1 commit into
mavlink:masterfrom
Mattbusel:fix/try-recv-concurrent-send
Open

Mattbusel wants to merge 1 commit into
mavlink:masterfrom
Mattbusel:fix/try-recv-concurrent-send

Conversation

@Mattbusel

@Mattbusel Mattbusel commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #570.

TCP and UDP readers and writers are clones of one socket, so they share its nonblocking flag. try_recv turned that flag on while holding only the reader lock. A send running at the same time under backpressure could then get WouldBlock, and on TCP that can happen after part of a frame is already on the wire.

Fix: SyncTransport's set_nonblocking hook is replaced by one method, try_read, which runs the read for try_recv. The default just reads, so serial and file are unchanged. TCP and UDP override it and switch the flag through their writer guard, so the flag can only change while no send is running, and the generic connection code no longer needs to know which transports share a socket.

Why not a read timeout: it would leave writes alone, but the shortest SO_RCVTIMEO costs about 8 ms per empty try_recv on Linux and 15.6 ms on Windows, against about 1 us for a nonblocking read. That breaks the "returns immediately" contract.

Tests in mavlink/tests/tcp_try_recv_concurrent_send_test.rs:

  • 20,000 FILE_TRANSFER_PROTOCOL sends while another thread loops on try_recv, with the server pausing 500 ms so buffers fill. Without the fix send 616 fails with WouldBlock (os error 10035 on Windows).
  • 100 empty try_recv calls return within 200 ms, and a following recv still waits for the 100 ms tcpout read timeout, so the socket is left blocking.
  • A frame written in two pieces returns WouldBlock until the rest arrives, then decodes intact.

Results: cargo test --workspace --features "dialect-all transport-tcp transport-udp std mav2-message-signing" passes on Linux (WSL Ubuntu 24.04) and the lib and integration tests pass on Windows. cargo clippy -p mavlink-core --all-targets is clean.

@onur-ozkan onur-ozkan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment thread mavlink-core/src/connection/tcp/sync.rs Outdated
Some(self.writer.lock().unwrap())
}

fn lock_writer_for_nonblocking_read(&self) -> Option<std::sync::MutexGuard<'_, Self::Writer>> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating another lock function next to lovk_writer seems very weak design choice to me. We should do something better than this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I dropped both hooks. SyncTransport now has one method, try_read, that runs the read for try_recv; TCP and UDP override it and switch the nonblocking flag through their writer guard, so the generic code no longer knows about the shared socket and the flag can't change while a send is running. I tried a short read timeout first so writes would be untouched, but the shortest one costs about 8 ms per empty try_recv on Linux and 15.6 ms on Windows versus about 1 us nonblocking, which doesn't fit a try_ call. I added tests for an empty try_recv returning fast and leaving the socket blocking, and for a frame split across two reads, and the workspace tests pass on Linux and Windows.

TCP and UDP readers and writers are clones of one socket and share its
nonblocking flag. try_recv flipped that flag with only the reader lock
held, so a concurrent send under backpressure could fail with WouldBlock,
for TCP after part of a frame was already written.

SyncTransport's set_nonblocking hook is replaced by a single try_read
method that runs the read for try_recv. The default reads as before.
TCP and UDP switch the flag through their writer guard, so it can only
change while no send is running, and the generic connection code no
longer needs to know which transports share a socket.

A short read timeout instead of nonblocking mode would leave writes
alone, but the shortest one costs about 8 ms per empty try_recv on Linux
and 15.6 ms on Windows, against about 1 us for a nonblocking read.

Tests: 20,000 FILE_TRANSFER_PROTOCOL sends while another thread polls
try_recv (failed on send 616 with WouldBlock before this change), 100
empty try_recv calls return within 200 ms and leave the socket blocking,
and a frame written in two pieces is decoded intact.

Fixes mavlink#570
@Mattbusel
Mattbusel force-pushed the fix/try-recv-concurrent-send branch from e278e89 to 1516385 Compare November 17, 2026 05:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

try_recv() can cause concurrent sends to fail

2 participants