sort: deduplicate file descriptors in merge mode by nonontb · Pull Request #11961 · uutils/coreutils

nonontb · 2026-04-23T15:32:48Z

What This Does

This PR makes sort -m (merge mode) use less (minimum?) opened files.

The Problem

Before:

If you ran sort -m file.txt file.txt file.txt, the program opened file.txt three times eagerly — once for every time it appeared on the command line.
With lots of duplicates or a tight system limit on open files, this could fail.

If you tried to merge a file that was also your output file, the program had to create a temporary copy behind the scenes, using one more file.

GNU version has no issue running the test in #5714

The Fix

Now the program opens each unique file only once and Lazily and use Mmap (memmap2 - unsafe) to manage one FD for all input file duplicates including re-use of output file as inputs.

Result

Fix #5714

github-actions · 2026-04-23T15:42:54Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/symlink (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/misc/write-errors was skipped on 'main' but is now failing.

codspeed-hq · 2026-04-23T16:13:13Z

Merging this PR will not alter performance

✅ 319 untouched benchmarks
⏩ 46 skipped benchmarks¹

_{Comparing nonontb:feature/sort-input-fd-optimization (59e06a5) with main (7f00530)}

46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

sylvestre · 2026-04-23T21:57:55Z

@@ -0,0 +1,3 @@
+1


please generate the files on the fly

sylvestre · 2026-04-26T09:19:18Z

@@ -0,0 +1,6 @@
+1


please generate this one the fly too

Yes, I forgot to delete it but the test is "on the fly"

#[test] fn test_merge_mixed_stdin_and_files() { let (at, mut ucmd) = at_and_ucmd!(); at.write("merge_duplicates_1.txt", "1\n3\n5\n"); // Verify that sort -m allows mixing stdin with files (GNU Coreutils compatible) ucmd.arg("-m") .arg("-") .arg("merge_duplicates_1.txt") .pipe_in("apricot\nelderberry\nkiwi\n") .succeeds() .stdout_is("1\n3\n5\napricot\nelderberry\nkiwi\n"); }

sylvestre · 2026-04-26T09:20:10Z

+    // it gets opened for writing. This allows reading the original content
+    // via memory-map while writing to the same file, without needing a temp copy.
+    let output_as_input = if let Some(name) = output.as_output_name() {
+        let output_path = Path::new(name).canonicalize()?;


maybe move this into a function?

sylvestre · 2026-04-26T09:21:20Z

did you look if we have a benchmark covering this? thanks

nonontb · 2026-04-26T12:29:15Z

did you look if we have a benchmark covering this? thanks

I just did and it seems there not relevant benchmark test in src/uu/sort/benches/sort_bench.rs even there is some doc to explain how to do it in src/uu/sort/BENCHMARKING.md or I miss something.

I suppose it would be better to add this bench test in another issue to have some reference numbers before benchmarking this PR ?

nonontb · 2026-04-26T17:31:53Z

did you look if we have a benchmark covering this? thanks

I just did and it seems there not relevant benchmark test in src/uu/sort/benches/sort_bench.rs even there is some doc to explain how to do it in src/uu/sort/BENCHMARKING.md or I miss something.

I suppose it would be better to add this bench test in another issue to have some reference numbers before benchmarking this PR ?

PR #12021

nonontb · 2026-04-27T17:26:53Z

Run the benchmark:

main:

Timer precision: 26 ns
sort_bench_merge           fastest       │ slowest       │ median        │ mean          │ samples │ iters
╰─ merge_pre_sorted_files  9.209 ms      │ 102.4 ms      │ 9.962 ms      │ 10.91 ms      │ 100     │ 100

This branch:

Timer precision: 25 ns
sort_bench_merge           fastest       │ slowest       │ median        │ mean          │ samples │ iters
╰─ merge_pre_sorted_files  9.243 ms      │ 99.39 ms      │ 9.981 ms      │ 11.18 ms      │ 100     │ 100

More or less the same

nonontb · 2026-04-30T10:10:19Z

The last failing test is not related to sort and code involved in this PR

nonontb · 2026-05-15T11:02:59Z

Hello @sylvestre,

Is this PR ready to merge on your side ?
It will fix the last fail test on the sort utility: sort-merge-fdlimit.

Plez, tell me if you need me to do more work on this ?

sylvestre · 2026-05-16T20:41:15Z

sorry for the latency

sylvestre · 2026-05-16T20:44:23Z

+            error,
+        })?;
+        // SAFETY: We keep the read_fd open for the lifetime of the memory-map,
+        // and we only read from it. The file is not modified while the


Are you sure? Another process could modify the file between the mmap and when it's read. This is a known mmap hazard - it won't cause UB in practice on Linux (you'd just read stale/partial data), but the safety comments should acknowledge this rather than claiming the file isn't modified.

No, you're right! the comment is misleading.
What I meant is the file won't be modified by the current sort process as writing to the output file won't happen during the read part of the current process.
I'll amend the comments => Tell me what you think

sylvestre · 2026-05-25T20:17:08Z

jobs are still failing

sylvestre · 2026-05-28T13:01:56Z

@@ -0,0 +1,621 @@
+// This file is part of the uutils coreutils package.


this large file is hard to review :(

Yes !
I will go through it again and try to reduce it

New tests are added for merging duplicate files.

… alias Mmap as MemoryMap

nonontb mentioned this pull request Apr 23, 2026

sort opens too many files #5714

Open

sylvestre reviewed Apr 23, 2026

View reviewed changes

nonontb force-pushed the feature/sort-input-fd-optimization branch 5 times, most recently from a9f249e to 3517780 Compare April 26, 2026 08:14

sylvestre reviewed Apr 26, 2026

View reviewed changes

nonontb mentioned this pull request Apr 26, 2026

Add benchmark for sort -m #12021

Closed

nonontb force-pushed the feature/sort-input-fd-optimization branch from 563ddcf to 8c39166 Compare April 27, 2026 17:24

nonontb force-pushed the feature/sort-input-fd-optimization branch from 3b51fcf to ff325ce Compare April 29, 2026 09:26

nonontb force-pushed the feature/sort-input-fd-optimization branch from ff325ce to 0f0942d Compare May 1, 2026 17:29

nonontb requested a review from sylvestre May 15, 2026 11:03

sylvestre reviewed May 16, 2026

View reviewed changes

Comment thread src/uu/sort/src/sort_input.rs Outdated

sylvestre reviewed May 16, 2026

View reviewed changes

Comment thread tests/by-util/test_sort.rs Outdated

sylvestre reviewed May 16, 2026

View reviewed changes

sylvestre reviewed May 28, 2026

View reviewed changes

nonontb force-pushed the feature/sort-input-fd-optimization branch from ab02b32 to 0980e3e Compare May 28, 2026 16:53

nonontb closed this May 28, 2026

nonontb force-pushed the feature/sort-input-fd-optimization branch from 0980e3e to 55549fa Compare May 28, 2026 17:36

nonontb added 15 commits May 28, 2026 19:39

sort: deduplicate FDs and defer opening unique files during merge

897dd2a

New tests are added for merging duplicate files.

chore: Update fuz cargo.lock

89e6445

chore: Run cargo fmt

2327e19

chore: Fix cargo clippy feedback

a832569

tests: Create test files on the fly

2255d29

tech: Fix use of unwrap(false)

a2a482c

fix: csspell - Rename Mmap to FileRead, use memory-map in comment and…

414ee46

… alias Mmap as MemoryMap

fix: Delete unused files in test

72f92c6

Refactor: Extract ouput-as-input as a function

5dab8b1

Fix: Failed test du to partial renaming

9ad58dc

fix: Ignore merge.rs test as WASI do not support temp dir

b9c8b07

fix: Add Mmap and variant to workspace.wordlist for cspell check

b272a05

fix: Remove print statement

0b4b64f

Fix: Replace AtomicUsize by primitive usize

c4ca17a

doc: Amend //SAFETY comment

4657b0a

nonontb reopened this May 28, 2026

Split and move sort_inputs.rs into sort_inputs/ module

59e06a5

		@@ -0,0 +1,621 @@
		// This file is part of the uutils coreutils package.

Uh oh!

Conversation

nonontb commented Apr 23, 2026

Uh oh!

github-actions Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nonontb Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sylvestre commented Apr 26, 2026

Uh oh!

nonontb commented Apr 26, 2026

Uh oh!

nonontb commented Apr 26, 2026

Uh oh!

nonontb commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nonontb commented Apr 30, 2026

Uh oh!

nonontb commented May 15, 2026

Uh oh!

sylvestre commented May 16, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sylvestre commented May 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 23, 2026 •

edited

Loading

codspeed-hq Bot commented Apr 23, 2026 •

edited

Loading

nonontb Apr 26, 2026 •

edited

Loading

nonontb commented Apr 27, 2026 •

edited

Loading