Batch UDP send with sendmmsg to cut per-packet syscall overhead (server)

[mcfnord]

Split out from the profiling discussion in #3662 (the MixEncodeTransmitData refactor thread) at @ann0see's request. This is the UDP-send finding from that profiling — out of scope for the mixer refactor, but a larger lever.

Context

While profiling the headless server for the mixer vectorization work (#3662 / #3739), the bigger surprise was where the rest of the CPU goes. Once the mix loop is vectorized, a large share of server CPU is the UDP socket path, not the audio code — and the send side scales with packet count × client count.

The server issues one sendto per client per frame: about N sends per frame for N connected clients. The payloads are tiny (tens to a few hundred bytes), so the fixed per-syscall cost (user/kernel mode switch, socket lookup, UDP framing) dominates the actual data movement. At N=50 that is ~50 send syscalls per frame, tens of thousands per second, most of it fixed overhead that grows linearly with the number of clients.

Proposal

Batch the per-frame outgoing packets with the Linux sendmmsg(2) call. The server already produces one outgoing packet per client per frame in CServer::MixEncodeTransmitData; instead of sending each one immediately, queue them and emit the whole frame with a single sendmmsg at the end of OnTimer. Each mmsghdr carries its own destination msg_name, so the different client addresses are handled in one call. This turns N send syscalls per frame into 1.

Measured result (working prototype)

I built a prototype and measured it. Batch storage + one sendmmsg flush per frame at the CSocket layer; CChannel::PrepAndSendPacket routes server sends into the batch, the client stays per-packet, and non-Linux platforms fall back to the existing per-packet path (compile-guarded).

Load = 20 synthetic tone clients, server pinned to 1 core (AMD EPYC 7R13, Zen3), Linux, GCC 13.3, on top of the vectorized mixer (#3739). Metric = server-process task-clock (all threads) over a fixed 40 s window of steady load; 6 interleaved runs per build; syscall counts from perf tracepoints.

build	server CPU / 40 s	as CPUs	send syscalls / 40 s
baseline (#3739)	12.50 s ± 0.20	0.313	152,986 sendto, 0 sendmmsg
+ sendmmsg batch	11.12 s ± 0.11	0.278	2,980 sendto, 15,000 sendmmsg

• Server CPU −11% at 20 clients (1.38 s less CPU per 40 s of load). The two arms do not overlap: min baseline > max batch, gap ≈ 7× the baseline sd.
• Audio send syscalls 150k → 15k (10× fewer) — exactly one sendmmsg per 2.67 ms frame. The ~3k residual sendto are non-audio protocol/ping sends, left per-packet on purpose.
• Load identical (20 connected every run); audio is byte-identical (the same packet bytes go to the same destinations, only the syscall boundary moves to end-of-frame).

A follow-up profile of the batched build confirms the mechanism: the send path drops to ~6% of server CPU (FlushSendBatch → a single __sys_sendmmsg per frame), and Opus encode/decode becomes the dominant remaining cost.

The saving grows with client count — the collapse factor is ~N, so a 50-client server sees a bigger cut than the 10× here. This is a larger lever than the mixer work because it scales with the thing that hurts big servers.

Scope / caveats

• sendmmsg is Linux-specific. macOS and Windows keep the current per-packet path via a compile guard — no behaviour change there. The Linux server (the common self-hosted / public-server case) is what benefits.
• Touches CSocket (batch buffer + flush), the one send call site in CChannel::PrepAndSendPacket, and one flush call in CServer::OnTimer. No audio math changes, so bit-exactness is not a concern; the risk surface is socket handling (partial sends, EINTR, per-message errors in the mmsghdr array — the prototype handles the first two and drops the tail of a frame on a hard per-message error, which UDP / the jitter buffer already tolerate).
• Receive batching with recvmmsg is a possible follow-up, but the current receive loop already drains the socket to EAGAIN per wakeup, so the send side is the clear first win. Left out of this change to keep it small.

I have the patch and the perf harness; happy to open a PR against this, or iterate on the design here first.

[ann0see]

Ok. The question is if batching adds latency. I'd assume that this is very likely but would of course need some more in depth measurements then. Maybe a toggle like multithreding. But please do NOT open any PRs. We need to get the design right first and then decide on possible approaches.

[ann0see]

Also mixing is O(n*n) while sending seems to be O(n) so asymptotically mixing is worse - I suppose. That syscals are slow isn't too surprising though as they need a context switch

[ann0see]

Anyway this means that a performance review of the new socket code is useful also.

[mcfnord]

Scaling measurement, since "grows with client count" deserved data rather than assertion. Same harness (interleaved A/B, server pinned to 1 core, perf task-clock over a fixed 40 s window of steady load), sweeping the client count:

clients	server CPU, baseline	server CPU, batched	CPU saved	send syscalls / 40 s: baseline → batched
10	0.181 CPUs	0.159 CPUs	12.4%	76,480 → 15,000 (5.1×)
20	0.316 CPUs	0.267 CPUs	15.5%	152,965 → 15,000 (10.2×)
40	0.634 CPUs	0.499 CPUs	21.3%	305,931 → 15,000 (20.4×)

Both axes grow with N. The batched build issues a constant ~15,000 send syscalls — one sendmmsg per 2.67 ms frame — no matter how many clients, while the per-packet baseline grows linearly (76k → 306k). So the CPU saving nearly doubles from 12% at 10 clients to 21% at 40, and it's still climbing: the server core is only ~63% utilized at N=40, so this isn't a saturation artifact. A 64-client server would fold ~32 sendto per frame into one call.

So the win is largest exactly where it matters most — the busy public servers. Load was identical per pair (connected count matched every run) and audio is byte-identical; only the send syscall boundary moves to end-of-frame.

[mcfnord]

Three answers to the points above — latency, receive batching, and the O(n²) question.

Latency (the main concern). You're right it's non-zero, so I measured it. The batch holds each frame's packets from when they're produced until one flush at the end of OnTimer, so I instrumented exactly that window (start of the per-client mix/encode loop → flush), steady state, same 1-core harness:

clients	mean window	tail max
20	0.40 ms	~1.0 ms
40	0.68 ms	~1.8 ms

That window is the worst case, and only for the first-encoded client — its packet waits the whole window, while the last-encoded client waits ~0, so the average added latency across clients is about half the window (~0.2 ms at 20, ~0.34 ms at 40). And it's bounded by the frame period by construction: OnTimer has to finish inside 2.67 ms or the server falls behind, so a packet can never be held past its own frame — no reordering, no skipped frame. Even the 40-client tail (1.8 ms) stays under 2.67 ms. Against real client RTT (tens of ms) and the jitter buffer (20–40 ms+), it's in the noise.

A runtime toggle like --multithreading is easy to add if you'd still prefer opt-in — happy to gate it that way for a conservative default.

Receive batching — I tried recvmmsg too, and it's not worth it. Interleaved A/B, both builds already batching sends, so the delta is purely the receive change:

clients	send-batch only	+ recvmmsg	Δ
20	0.277 CPUs	0.283	−2.2%
40	0.503 CPUs	0.500	+0.7%

Receive syscalls dropped ~3× (213k → 71k at N=20), but datagrams arrive spread out in time, so each recvmmsg only pulls ~3 at a time and the per-call overhead (rebuilding the message array + a copy into the shared receive buffer) eats the saving. The existing drain-to-EAGAIN loop is already cheap. So I'd keep the change to sendmmsg only — smaller surface, and that's where the win actually is.

O(n²) mixing vs O(n) sending. Agreed, asymptotically mixing is worse. But in the range real servers run — profile at 20 clients, 1 core — mixing self-time is ~3.4%, while the send path was the single largest bucket before batching. So the crossover is well above today's loads, which argues for doing both (the mixer refactor in #3662 and the send batching), not either/or. The send batch is just the bigger lever at current client counts and it's a small, contained change.

Holding on the PR per your note — happy to keep this at the design level. Anything you'd want measured next?

[ann0see]

From a strict latency standpoint every increase in latency would be unacceptable. So not sure if batching is a way to go then.

[pljones]

Yeah - we have to deal with 150 client servers that might not be multi-threaded. If this would push them over the frame period, it's broken.

By the look of it, it's never more responsive and the CPU saving - in absolute terms rather than percentage - isn't particularly significant. 60 clients goes from 0.634 of a core to 0.499 of a core. Doubling to 120 and you're maxing out that core either way. (I guess the 150 client single threaded server's a write off, then...)