Skip to content

add udp-dont-frag option to support force setting DF bit - #242

Open
dust-li wants to merge 1 commit into
Mellanox:sockperf_v2from
dust-li:sockperf_v2
Open

add udp-dont-frag option to support force setting DF bit#242
dust-li wants to merge 1 commit into
Mellanox:sockperf_v2from
dust-li:sockperf_v2

Conversation

@dust-li

@dust-li dust-li commented Sep 30, 2025

Copy link
Copy Markdown

When measuring maximum packets-per-second (PPS) on Linux using multiple sockperf processes to stress UDP throughput, the kernel’s per-packet atomic operation for generating IP ID becomes a bottleneck.

Setting the Don’t Fragment (DF) bit in IP headers allows the kernel to skip this atomic IP ID assignment (as DF packets can use a constant or zero ID), reducing contention and improving scalability under high load.

  • Testing:
for((i=0;i<64;i++)); do
    taskset -c $i sockperf tp -i 200.16.10.2 -t 100 --udp-dont-frag
done
  • perf top without udp-dont-frag
    56.56% [kernel] [k] ip_idents_reserve
    5.98% [kernel] [k] __dev_xmit_skb
    3.73% [kernel] [k] skb_set_owner_w
    2.37% [kernel] [k] mlx5e_xmit
    2.08% [kernel] [k] mlx5e_consume_skb
    1.63% [kernel] [k] mlx5e_poll_tx_cq
    1.43% [kernel] [k] mlx5e_handle_tx_dim
    1.28% [kernel] [k] sock_wfree
    1.12% [kernel] [k] __slab_free
    0.64% [kernel] [k] build_detached_freelist
    0.61% [kernel] [k] fib_table_lookup
    0.59% sockperf [.] client_handler<IoRecvfrom, SwitchOff, PongModeNever>
    0.57% [kernel] [k] pfifo_fast_dequeue
    0.51% [kernel] [k] udp_sendmsg
    0.50% libpthread-2.32.so [.] __libc_sendto
    0.50% [kernel] [k] __ip_append_data

  • Without this option:
    05:23:49 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
    05:23:50 PM lo 22.00 22.00 8.61 8.61 0.00 0.00 0.00 0.00
    05:23:50 PM eth0 2.00 7958649.00 0.20 466327.09 0.00 0.00 1.00 1.91
    05:23:50 PM eth1 1.00 6588679.00 0.12 386055.41 0.00 0.00 1.00 1.58
    05:23:50 PM bond0 3.00 14547313.00 0.32 852381.62 0.00 0.00 2.00 1.75

  • With this option:
    Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
    Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    Average: eth0 1.00 12339901.50 0.10 723041.10 0.00 0.00 0.50 2.96
    Average: eth1 1.00 13850533.00 0.10 811554.67 0.00 0.00 0.50 3.32
    Average: bond0 2.00 26190604.50 0.20 1534605.73 0.00 0.00 1.00 3.14

When measuring maximum packets-per-second (PPS) on Linux using multiple
sockperf processes to stress UDP throughput, the kernel’s per-packet atomic
operation for generating IP ID becomes a bottleneck.

Setting the Don’t Fragment (DF) bit in IP headers allows the kernel to
skip this atomic IP ID assignment (as DF packets can use a constant or
zero ID), reducing contention and improving scalability under high load.

- Testing:
```
for((i=0;i<64;i++)); do
    taskset -c $i sockperf tp -i 200.16.10.2 -t 100 --udp-dont-frag
done
```

- perf top without udp-dont-frag
  56.56%  [kernel]                                [k] ip_idents_reserve
   5.98%  [kernel]                                [k] __dev_xmit_skb
   3.73%  [kernel]                                [k] skb_set_owner_w
   2.37%  [kernel]                                [k] mlx5e_xmit
   2.08%  [kernel]                                [k] mlx5e_consume_skb
   1.63%  [kernel]                                [k] mlx5e_poll_tx_cq
   1.43%  [kernel]                                [k] mlx5e_handle_tx_dim
   1.28%  [kernel]                                [k] sock_wfree
   1.12%  [kernel]                                [k] __slab_free
   0.64%  [kernel]                                [k] build_detached_freelist
   0.61%  [kernel]                                [k] fib_table_lookup
   0.59%  sockperf                                [.] client_handler<IoRecvfrom, SwitchOff, PongModeNever>
   0.57%  [kernel]                                [k] pfifo_fast_dequeue
   0.51%  [kernel]                                [k] udp_sendmsg
   0.50%  libpthread-2.32.so                      [.] __libc_sendto
   0.50%  [kernel]                                [k] __ip_append_data

- Without this option:
  05:23:49 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
  05:23:50 PM        lo     22.00     22.00      8.61      8.61      0.00      0.00      0.00      0.00
  05:23:50 PM      eth0      2.00 7958649.00      0.20 466327.09      0.00      0.00      1.00      1.91
  05:23:50 PM      eth1      1.00 6588679.00      0.12 386055.41      0.00      0.00      1.00      1.58
  05:23:50 PM     bond0      3.00 14547313.00      0.32 852381.62      0.00      0.00      2.00      1.75

- With this option:
  Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
  Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
  Average:         eth0      1.00 12339901.50      0.10 723041.10      0.00      0.00      0.50      2.96
  Average:         eth1      1.00 13850533.00      0.10 811554.67      0.00      0.00      0.50      3.32
  Average:        bond0      2.00 26190604.50      0.20 1534605.73      0.00      0.00      1.00      3.14

Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
@svc-nbu-swx-media

Copy link
Copy Markdown

Can one of the admins verify this patch?

@dust-li

dust-li commented Nov 13, 2025

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants