dnsdist closes congested DNS-over-QUIC streams with INTERNAL_ERROR

[Spiffyk]

• This is not a support question, I have read about opensource and will send support questions to the IRC channel, Github Discussions or the mailing list.
• I have read and understood the 'out in the open' support policy

• Program: dnsdist
• Issue type: Bug report

Short description

While testing my implementation of DNS-over-QUIC in DNS Shotgun against dnsdist, I have found that it closes a large number of streams with an INTERNAL_ERROR app code.

Environment

• Operating system: OpenSUSE Tumbleweed
• Software version: master - 5012710
• Software source: Compiled myself

Steps to reproduce

1. Send high-volume DNS-over-QUIC traffic to dnsdist, e.g. using DNS Shotgun.

Expected behaviour

Pretty much 100% of the queries should be answered in a localhost setting.

Actual behaviour

A large number of the queries are lost because dnsdist closes the streams with an INTERNAL_ERROR app code.

Other information

After inspecting the code, the culprit seems to be that the quiche_conn_stream_send function returns the value -1 (QUICHE_ERR_DONE), which is because the stream is congested, as per the Quiche documentation. This causes dnsdist to lose a lot of legitimate queries in a high-volume traffic scenario.

When quiche_conn_stream_send returns QUICHE_ERR_DONE, dnsdist should defer sending the reply to a later time at which it gets notified that the stream is writable again.

[rgacogne]

Hi! Thank for reporting this! @chbruyand has noticed this as well, we are looking into it.

[rgacogne]

@Spiffyk I'm currently testing a patch that should fix the issue, and I was wondering if I could easily reproduce your tests? I can build DNS shotgun from your branch, but is there a way to generate similar PCAP(s) to the one(s) you are using?

[Spiffyk]

@rgacogne I use actual captures and can't share those PCAPs, but there is a pcap/generate-const-qps.lua script that I think should do the job.

[rgacogne]

Understood, I'll look into it. Thanks!

[Spiffyk]

INTERNAL_ERROR is gone, but I'm still getting some strange results, especially on long-lived connections.

The chart above contains latencies - a test of Shotgun DoQ in comparison with DoT against dnsdist (commit af62fb1), on pure localhost. Dnsdist is set to send a REFUSED RCode for everything (to simply test baseline performance of QUIC) using addAction(AllRule(), RCodeAction(DNSRCode.REFUSED)).

By default, Shotgun reuses connections and sends multiple queries over them. In a no-idle scenario, Shotgun creates a new connection for each query. Everything on the 2000 milliseconds mark has been lost (2000 ms is the default timeout). Note that the chart uses a logarithmic scale.

For reference, here is a chart when tested against a DoQ echo server written in Go under the same conditions (without DoT because this server is DoQ-only):

Shall I open another issue?

[rgacogne]

Thanks! So there is clearly a bug somewhere in our stack. Given that this issue has not been referenced in a release/changelog yet I'm just going to re-open it.

[rgacogne]

I believe this is now fixed, but please let me know if you think otherwise!