ringdl2

A lightweight, single-threaded Linux downloader built in Rust that leverages modern io_uring, O_DIRECT, and unbuffered streaming TLS to minimize total RAM usage (RSS and page cache) and total CPU usage (user + system) without compromising on download speed.

---

1. Overview & Motivation

File downloading utilities (like aria2) have become very popular because they are lightweight (with respect to CPU and RAM usage) and support a variety of file downloading protocols like HTTPS, BitTorrent, FTP, etc.

Tools like aria2 were built to support older Linux kernels alongside platforms like Windows and macOS. On Linux, aria2 largely uses standard POSIX network and storage I/O along with an epoll-based event loop. This is a battle-tested, efficient design that explains its widespread adoption.

However, modern Linux kernels (5.1+) with io_uring offer a unified, asynchronous interface across both networking and block storage. This is a promising fit for a file downloader's event loop and I/O pipeline. The reduction in system calls and ability to unify socket reads and disk writes in a single ring buffer can offer advantages over traditional syscall-heavy POSIX architectures.

In a conventional downloader architecture (outside of niche kernel bypass mechanisms), encrypted download data passes through a multi-stage pipeline involving several buffer boundaries:

1. Network packets arrive into kernel socket buffers (sk_buff).
2. The runtime copies ciphertext from kernel space to user space via read() / recv().
3. A TLS library decrypts ciphertext into an intermediate user-space buffer.
4. Plaintext is copied from the user buffer into the Linux page cache via POSIX write().
5. Kernel writeback threads asynchronously copy dirty pages from the page cache to physical storage media.

For bulk sequential downloads (e.g., multi-gigabyte datasets, disk images, archives), this standard path incurs CPU overhead from repeated memory copies, per-syscall context transitions, and page cache pressure from large Inactive(file) residency.

ringdl2 explores a unified io_uring architecture that eliminates intermediate staging buffers and bypasses the page cache.

Disclaimer: ringdl2 is currently a feature-limited MVP built specifically to gather performance data for this architecture, using aria2 as a performance baseline. It exclusively supports HTTPS file downloading for this testing. This MVP is not intended for general-purpose use beyond this scope at this time.

---

2. Architecture & Data Flow

+---------------------------------------------------------------------------------------+
|                              Pinned Memory Pool (Registered)                          |
|                                                                                       |
|  +-----------------------+     In-Place Decryption & Compaction +------------------+  |
|  | Dedicated 1MB         | -----------------------------------> | 4KB-Aligned 1MB  |  |
|  | Read Block (Cipher)   |     (rustls unbuffered streaming)    | Write Block Pool |  |
|  +-----------------------+                                      +------------------+  |
+-----------^---------------------------------------------------------------|-----------+
            |                                                               |
   IORING_OP_READ_FIXED                                            IORING_OP_WRITE_FIXED
   (Kernel -> Userspace)                                            (O_DIRECT -> Disk)
            |                                                               |
      [TCP Socket]                                                     [NVMe / SSD]

The Pinned Memory Pool

• Structure: A single contiguous virtual memory allocation (16MB–256MB), pinned in RAM via mlock and registered with the kernel via IORING_REGISTER_BUFFERS.
• Alignment: Aligned to 4KB (4,096 bytes) boundaries to satisfy O_DIRECT block I/O constraints.
• Partitioning: Divided evenly across connections. Each connection receives:
  • 1 Dedicated Read Block: Permanently assigned for receiving ciphertext from the socket.
  • Dynamic Write Pool: A set of blocks managed via an $O(1)$ hardware bitmask (trailing_zeros()) to hold decrypted plaintext and back in-flight disk writes.

Step-by-Step Data Path

1. Network Read (IORING_OP_READ_FIXED): The connection submits a fixed-buffer read SQE targeting its dedicated Read Block at the current read cursor. The kernel copies incoming TCP data into the Read Block.
2. In-Place Decryption & Compaction: When the read CQE completes, rustls unbuffered record processing parses and decrypts TLS frames in-place directly within the Read Block. Because TLS framing adds headers and authentication tags, the resulting plaintext is fragmented and unaligned. To satisfy O_DIRECT's strict 4KB sector alignment requirements and remove the garbage intermediaries, the decrypted plaintext is explicitly compacted (via selective copying) tightly into an active, disk-aligned Write Block checked out from the pool.
3. Direct Disk Write (IORING_OP_WRITE_FIXED + O_DIRECT): Once a Write Block reaches a 4KB-aligned boundary (or EOF), an asynchronous write SQE is submitted. O_DIRECT bypasses the kernel page cache, writing data directly to the storage device without populating Inactive(file) pages in RAM.
4. Natural Backpressure: If storage writes stall, the connection exhausts its Write Pool. Decryption pauses, the Read Block remains full, and the application stops issuing read SQEs. The TCP receive window closes, throttling the remote server via standard TCP flow control until disk write CQEs return and recycle Write Blocks.

Event Loop Mechanics

• Zero-Syscall Staging: New read and write SQEs across all connections are staged directly into the shared submission ring in user space.
• Single-Syscall Dispatch & Wait: A single io_uring_enter call submits all staged SQEs and suspends the thread if no completions are ready.
• Zero-Syscall CQE Harvesting: Completions are read directly from the shared completion ring without re-entering the kernel.

Memory Copy Accounting

Both pipelines share two unavoidable operations: a kernel-to-user copy for network ingress (io_uring zero copy for this is still in a hardware niche) and a cryptographic transform-copy for TLS decryption. The difference logically lies in the remaining pipeline stages:

Pipeline Stage	Conventional Downloader (aria2c)	ringdl2
Network Ingress	Kernel socket → User buffer (CPU copy)	Kernel socket → Pinned Read Block (CPU copy)
TLS Decryption	Ciphertext → Plaintext heap buffer (crypto transform)	In-place within Read Block (crypto transform)
Alignment & Compaction	N/A (handled by kernel page cache)	Read Block → Disk-aligned Write Block (CPU copy)
Storage Write	Plaintext buffer → Page Cache via write() (CPU copy)	Write Block → Disk via O_DIRECT (DMA, no page cache)
Storage Flush	Page Cache → Disk (async kernel writeback)	N/A (already on disk)

Design Limitations

• ringdl2 requires Linux Kernel 5.1+ whereas aria2c has fallback mechanisms for much older kernels.
• ringdl2 is strictly structured around io_uring, which inherently breaks compatibility with non-Linux environments supported by aria2c such as macOS and Windows.
• ringdl2 requires modern Linux filesystems like ext4, XFS, or Btrfs for O_DIRECT support, while aria2c supports a wider range of formats including exFAT.

---

3. Empirical Benchmarks: 10-Iteration Interleaved Study

To evaluate performance and memory subsystem characteristics, ringdl2 was benchmarked against aria2c in a strict 10-iteration interleaved A/B protocol (20 total runs) downloading a 1.0 GiB payload over HTTPS across 8 concurrent connections.

Environment

• Client System: Debian GNU/Linux 13 (trixie) VM (Kernel Linux 7.1.3+deb13-cloud-arm64 aarch64) on Apple MacBook Air M3 as host.
• Compiler: rustc 1.85.0
• Baseline Tool: aria2c version 1.37.0
• Server: Nginx 1.31 running in Docker on an external Linux node.
• Network: Server and client host are on the same private LAN with a router node in the middle.

Test Protocol

• Target: 1.0 GiB file containing random data served over HTTPS.
• Cache Purge: sync && echo 3 > /proc/sys/vm/drop_caches executed immediately prior to every run.
• Thermal Cooldown: 15-second pause between each execution.
• Metrics Collected: Exact pre/post delta from /proc/meminfo, process execution statistics from /usr/bin/time -v, and full SHA256 file integrity validation.

Build & Run

# Build release binary
cargo build --release

# Run ringdl2 (Usage: ringdl2 <URL> <DEST_DIR> <CONNECTIONS>)
# Requires root for mlock() of the pinned memory pool beyond default ulimits.
sudo ./target/release/ringdl2 "https://SERVER_IP:SERVER_PORT/test_1GB.bin" ./downloads 8

# Run aria2c baseline
aria2c --check-certificate=false -x 8 -s 8 -d ./downloads "https://SERVER_IP:SERVER_PORT/test_1GB.bin"

Results (Median ± $\sigma$)

Metric	aria2c (Median ± $\sigma$)	ringdl2 (Median ± $\sigma$)	ringdl2 Delta
Page Cache Inflation (Cached $\Delta$)	1,041.68 MB ± 5.30	5.07 MB ± 1.72	-1,036.61 MB (99.5% reduction)
Inactive File Cache $\Delta$	1,026.92 MB ± 2.12	1.45 MB ± 1.76	-1,025.47 MB
User CPU Time	1.58s ± 0.12	0.71s ± 0.08	-55.1% (2.23x faster)
System (Kernel) CPU Time	1.55s ± 0.21	1.25s ± 0.14	-19.7% (less syscall overhead)
Total CPU Time (User + Sys)	3.17s ± 0.31	1.98s ± 0.21	-37.7% less CPU time
Throughput (Network Speed)	636.3 Mbps ± 35.5	629.3 Mbps ± 25.7	Negligible
Wall-Clock Time	13.50s ± 0.91	13.65s ± 0.64	Negligible
Minor Page Faults	5,694 ± 747	1,006 ± 98	82.3% reduction
Major (I/O) Page Faults	22 ± 0	2 ± 0	90.9% reduction
Involuntary Context Switches	31 ± 7	15 ± 5	51.6% reduction
Voluntary Context Switches	42,399 ± 999	55,546 ± 1,505	+31.0% increase
Peak Resident Set Size (RSS)	24.65 MB ± 0.29	259.75 MB ± 0.06	+953.7% (10.5x larger)
SHA256 Match Rate	10 / 10 (100%)	10 / 10 (100%)	Bit-perfect integrity

The raw JSON dataset containing all individual run metrics is available in benchmarks_10iter.json.

---

4. Key Takeaways & Trade-offs

1. Memory Subsystem Trade-off

• aria2c maintains a small upfront user-space RSS (~25 MB) by delegating buffering to the kernel page cache. For a 1GB download, this populates ~1.03 GB of Inactive(file) memory in the OS page cache.
• ringdl2 allocates and pins a fixed user-space buffer pool upfront (~260 MB RSS via mlock), but restricts page cache inflation to ~5 MB (binary + filesystem metadata). On systems running concurrent workloads (databases, application servers), this prevents large file ingress from causing system-wide memory buildup.
• While it's technically correct that aria2 doesn't ever require O(file_size) in terms of resident memory, it can indirectly use up to O(file_size) in page cache if available. ringdl2 uses O(1) memory in comparison, regardless of the file size. It is important to clarify that page cache usage in aria2 isn't necessarily an anti-pattern. In fact, page caching is a desirable use of available RAM in general. However, ringdl2 slashes this usage, providing a cleaner memory footprint to both the administrator and the kernel.

2. CPU Efficiency

• Despite requiring an explicit user-space memory copy to pack and align plaintext for O_DIRECT, bypassing the kernel's write() memory copy and page cache traversal reduced user CPU time by 55.1% (0.71s vs 1.58s).
• Batched io_uring submission and completion harvesting lowered kernel system time by 19.7% (1.25s vs 1.55s).
• Pre-allocated, pinned memory buffers reduced minor page faults by 82.3% (1,006 vs 5,694).

3. Direct I/O Constraints

• O_DIRECT requires sector-aligned (4KB) memory addresses, file offsets, and write lengths.
• ringdl2 aligns all chunk boundaries to 4KB multiples and pads final EOF blocks to sector boundaries, relying on ftruncate for exact size finalization. Direct I/O requires underlying filesystem support (e.g., ext4, XFS, Btrfs).

---

Benchmark Setup Limitations

• The benchmark was run over a ~636 Mbps LAN link, which both tools fully saturated. Wall-clock times are therefore bandwidth-bounded, not CPU-bounded. CPU and memory subsystem differences would be more pronounced on faster links (10GbE+) where the CPU cost of memory copies and syscall overhead becomes the bottleneck rather than the wire.
• The test was conducted on an ARM64 VM client. It will be interesting to see performance on bare metal and x86 architectures.

---

6. Current Status & Roadmap

ringdl2 is MVP complete with io_uring + O_DIRECT + minimal-copy unbuffered TLS architecture. The benchmarks show ringdl2 consuming less total CPU and exhibiting a lower overall RAM footprint than aria2c while maintaining the same network throughput. ringdl2 is being considered for broader feature buildup and further core optimizations to develop as a more performant, fully-featured alternative to aria2c for modern Linux environments.