scx_soft_domain: through topology-aware scheduling to improve perf - #3612
Open
CyangCyang wants to merge 1 commit into
Open
scx_soft_domain: through topology-aware scheduling to improve perf#3612CyangCyang wants to merge 1 commit into
CyangCyang wants to merge 1 commit into
Conversation
sirlucjan
requested changes
Jun 12, 2026
sirlucjan
left a comment
sirlucjan
left a comment
Collaborator
There was a problem hiding this comment.
Could you move scheduler to: https://github.com/sched-ext/scx/tree/main/scheds/experimental please?
Contributor
|
Can you fix these build errors? Thanks. |
Collaborator
|
Still doesn't compile |
Author
|
Author
Since I'm working with a relatively old kernel version, a few APIs are not consistent with the latest upstream. I've just modified them. |
sirlucjan
requested changes
Jun 17, 2026
Collaborator
There was a problem hiding this comment.
lucjan at cachyos ~/Pobrane/scx/target/release 7:41:10 fd86c33f main
❯ sudo ./scx_soft_domain
[sudo] hasło użytkownika lucjan:
07:41:24 [INFO] libbpf: struct_ops soft_domain_ops: member sub_attach not found in kernel, skipping it as it's set to zero
07:41:24 [INFO] libbpf: struct_ops soft_domain_ops: member sub_detach not found in kernel, skipping it as it's set to zero
07:41:24 [INFO] libbpf: struct_ops soft_domain_ops: member sub_cgroup_id not found in kernel, skipping it as it's set to zero
07:41:24 [WARN] libbpf: map 'soft_domain_ops': BPF map skeleton link is uninitialized
07:41:24 [INFO] scx_soft_domain scheduler initialized
07:41:24 [INFO] scx_soft_domain scheduler started
^CReceived Ctrl+C, exiting...
All schedulers in the repo display the version number (and the Git commit ID), which makes debugging easier.
lucjan at cachyos ~/Pobrane/scx/target/release 7:41:46 fd86c33f main
❯ sudo scx_pandemonium
[07:42:21] [INFO] scx_pandemonium 5.12.0 x86_64-unknown-linux-gnu SMT on
[07:42:21] [INFO] CPUS: 16 (governor: performance)
[07:42:21] [INFO] VERBOSE: false
libbpf: struct_ops pandemonium_ops: member sub_attach not found in kernel, skipping it as it's set to zero
libbpf: struct_ops pandemonium_ops: member sub_detach not found in kernel, skipping it as it's set to zero
libbpf: struct_ops pandemonium_ops: member sub_cgroup_id not found in kernel, skipping it as it's set to zero
[07:42:22] [INFO] L2 GROUP 0: [0,8]
[07:42:22] [INFO] L2 GROUP 1: [1,9]
[07:42:22] [INFO] L2 GROUP 2: [2,10]
[07:42:22] [INFO] L2 GROUP 3: [3,11]
[07:42:22] [INFO] L2 GROUP 4: [4,12]
[07:42:22] [INFO] L2 GROUP 5: [5,13]
[07:42:22] [INFO] L2 GROUP 6: [6,14]
[07:42:22] [INFO] L2 GROUP 7: [7,15]
[07:42:22] [INFO] L2 GROUPS: 8 across 16 CPUs, 1 SOCKETS
[07:42:22] [INFO] LLC DOMAINS: 1 (SAME-CCX RUNG INACTIVE -- MONOLITHIC L3)
[07:42:22] [INFO] TOPOLOGY SPECTRUM: lambda2=16.0000 tau=10ms codel_eq=8000us
[07:42:22] [INFO] RESISTANCE AFFINITY: CPU 0 rank: CPU8(R=0.001), CPU1(R=0.063), CPU2(R=0.063)
[07:42:22] [INFO] RESISTANCE AFFINITY: R_eff L2=0.0010 non-L2=0.0630 ratio=63.4x
[07:42:22] [INFO] PANDEMONIUM IS ACTIVE (CTRL+C TO EXIT)
[07:42:22] [INFO] PROCDB: LOADED 4 PROFILES FROM /var/lib/pandemonium/procdb.bin
^C[07:42:27] [INFO] PROCDB: SAVED 4/33 PROFILES TO /var/lib/pandemonium/procdb.bin
[KNOBS] regime=MIXED slice_ns=1000000 batch_ns=15000000 preempt_ns=749969 mwu=0.996 ticks=L:0/M:5/H:0 frozen=0 l2_hit=B:95%/I:80%/L:0% xccx_scatter_pct=0 xccx_sel_tight=0 xccx_sel_sync=0 xccx_sel_normal=0 xccx_sel_dfl=0 xccx_enq_t1=0 xccx_enq_t2=0 xccx_steal=0 xccx_step5=0
[07:42:27] [INFO] PANDEMONIUM IS SHUTTING DOWN
PANDEMONIUM SUMMARY
TOTAL DISPATCHES: 2750
TOTAL IDLE HITS: 1190
TOTAL SHARED: 185
TOTAL PREEMPT: 0
TOTAL KEEP_RUN: 0
PEAK DISPATCH/S: 634
AVG DISPATCH/S: 687
IDLE HIT RATE: 43.3%
ELAPSED: 4.0s
SAMPLES: 5
[07:42:27] [INFO] Shutdown complete
lucjan at cachyos ~/Pobrane/scx/target/release 7:42:27 fd86c33f main
❯ sudo scx_cake
[2026-06-17T05:42:35Z WARN scx_cake] SCX_CAKE_ENQ_KICK_IDLE=1 but booted kernel lacks SCX_ENQ_KICK_IDLE; using explicit kick
[2026-06-17T05:42:36Z INFO scx_cake] IRQ-avoid: enabled but no IRQ-noisy CPUs detected
[2026-06-17T05:42:36Z INFO scx_cake] Topology Strategy: Per-CPU local-first dispatch
[2026-06-17T05:42:36Z INFO scx_cake] Core spread route table: top CPUs [(1, 232), (5, 232), (9, 232), (13, 232), (7, 226), (15, 226)]
[2026-06-17T05:42:36Z INFO scx_cake] libbpf: struct_ops cake_ops: member sub_attach not found in kernel, skipping it as it's set to zero
[2026-06-17T05:42:36Z INFO scx_cake] libbpf: struct_ops cake_ops: member sub_detach not found in kernel, skipping it as it's set to zero
[2026-06-17T05:42:36Z INFO scx_cake] libbpf: struct_ops cake_ops: member sub_cgroup_id not found in kernel, skipping it as it's set to zero
[2026-06-17T05:42:36Z WARN scx_cake] libbpf: map 'cake_ops': BPF map skeleton link is uninitialized
[2026-06-17T05:42:36Z INFO scx_cake] scx_cake 1.2.0 x86_64-unknown-linux-gnu SMT on
[2026-06-17T05:42:36Z INFO scx_cake] scheduler options: scx_cake
[2026-06-17T05:42:36Z INFO scx_cake] release accelerators: route-pred=off, confidence=off, llc-pending=on, local-waiter=on, domain-drr=off, trust-maps=off, core-steal-dhq=off
[2026-06-17T05:42:36Z INFO scx_cake] 16 CPUs, 1 LLCs, profile: gaming, quantum: 1000us, queue-policy: llc-vtime, storm-guard: shield, busy-wake-kick: policy, learned-locality: off, wake-chain-locality: off, core-steal-dhq: off
^Z
[1] + 64811 suspended sudo scx_cake
Could you implement this?
CyangCyang
force-pushed
the
main
branch
2 times, most recently
from
June 22, 2026 03:00
1851dd2 to
eb41c9d
Compare
Author
Thanks for the review and the suggestion. I've now implemented the display of both the version number and the Git commit ID in the scheduler startup message. |
sirlucjan
reviewed
Jun 22, 2026
sirlucjan
left a comment
sirlucjan
left a comment
Collaborator
There was a problem hiding this comment.
Is there a reason you're using older versions of anyhow, clap, ctrlc, libc, and log?
Collaborator
|
I ran a benchmark (https://github.com/CachyOS/cachyos-benchmarker), and scx_soft_domain didn't perform very well. It crashes a lot. First attempt: Second attempt: |
Collaborator
|
Third attempt: |
sirlucjan
requested changes
Jun 22, 2026
sirlucjan
left a comment
sirlucjan
left a comment
Collaborator
There was a problem hiding this comment.
So, please fix above stalls.
Author
Oh, apologies! I have revised the content. Here is the updated version for the review: Among the tests provided by cachyos-benchmarker, the workload patterns do not fully align with the optimisation model of this scheduler. Below are the test results: The scx_soft_domain scheduler is designed with Redis‑workloads in mind. It can optimise for CPU‑intensive services, and it performs especially well for workloads that suffer from cross‑LLC / cross‑NUMA memory accesses.
With scx_soft_domain, provided that CPU load is not saturated, each Redis instance remains pinned to a specific CPU without migration, and is additionally spread as evenly as possible across all NUMA nodes and their respective clusters.
Latency has dropped significantly across the board – average latency, P50, P95, and P99 all improved by over 65%, with particularly notable gains in P95 and P99 tail latency, which means more stable response times under high‑concurrency scenarios. It is worth noting that the current scheduler is specifically tuned for workloads that share the performance profile of Redis. In future updates, we plan to release additional optimisation policies for other workload types, such as MySQL‑like models. Thank you for your review! |
sirlucjan
requested changes
Jul 2, 2026
sirlucjan
left a comment
sirlucjan
left a comment
Collaborator
There was a problem hiding this comment.
Please update Cargo.lock
cargo update -w
Collaborator
|
You did not update the scx_cargo dependencies, etc., to the latest version. |
Author
Thanks! I've updated the dependency versions and rebased onto the latest main branch. Just pushed. |
Collaborator
|
The latest version is 1.1.2 |
Author
Oh sorry, I've just updated it and pushed again. Thanks! |
sirlucjan
requested changes
Jul 3, 2026
Contributor
|
Can you add all the details mentioned in the PR also in the commit message? It's a lot more convenient to see the details from git log rather than searching them on github. Thanks. |
CyangCyang
force-pushed
the
main
branch
2 times, most recently
from
July 4, 2026 02:09
065138f to
16c22b4
Compare
Author
Thanks for the suggestion! I've amended the commit message to include all the details from the PR description. |
Collaborator
Not bad! |
sirlucjan
previously approved these changes
Jul 6, 2026
Author
|
I found the CI gate logs triggered 3 days ago in Actions and saw the following error reported by the veristat job. I have now fixed this BPF security verification issue and successfully verified it with veristat in my personal repository. |
CyangCyang
force-pushed
the
main
branch
3 times, most recently
from
July 10, 2026 08:58
bdab0a4 to
5d9cbfc
Compare
sirlucjan
self-requested a review
July 10, 2026 19:49
Collaborator
|
Collaborator
|
Unfortunately, I've noticed that scx_soft_domain causes very noticeable clipping and audio crackling under almost every possible load. |
Author
I just ran some tests on my end to try to reproduce the noticeable clipping and audio crackling you described, but I wasn't able to replicate the issue under my test conditions. It might be system- or machine-specific, as hardware topology and kernel configurations can vary significantly. For reference, my test environment consists of 4 NUMA nodes, 96 CPUs, running Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, with VERSION_CODENAME=resolute). During my testing, I also benchmarked the existing schedulers already available in the repository for comparison. I've attached the results chart below for your reference. In my view, different schedulers suit different use cases, not every scheduler will excel in every test. 
|
Author
Hi, could you please share your machine's topology information with me? I'd like to try reproducing the results from your tests and then fix the issue. |
Author
|
Hi, The fix has now been applied, and the test results after the fix are as follows: |
…ance Background In the performance test of multi-instance applications, such as Redis, a large number of CPU migrations across clusters or NUMA nodes are detected by capturing cputrace. In addition, there is cross-NUMA memory access. In addition, multiple Redis processes on the same CPU are switched back and forth. When the test pressure is high, the CPU competition is fierce. We hope to reduce the Redis migration and competition and reduce the cross-NUMA memory access based on the topology information. Introduce scx_soft_domain is a scheduler dedicated to multi-instance services, such as Redis. It detects the CPU hardware topology (LLC and NUMA) and task affinity, and executes tasks on the CPUs in the domain where the tasks are located as much as possible. This reduces data transmission across LLCs and NUMAs, improves cache locality, and improves the overall system performance. Core implementation logic The CPU selection policy is implemented as follows: Multiple instances are evenly distributed on the machine. Within the same scheduling domain (that is, the same LLC or NUMA node), tasks are preferentially scheduled to CPUs in the domain, and cross-domain migration is avoided as much as possible. When the load is high, load balancing is performed. First, whether the task can be directly executed on the selected CPU is quickly determined based on the load of the selected CPU and the running status of the task on the selected CPU. If the task cannot be directly executed on the selected CPU, the task is inserted into a user-defined queue of the LLC where the selected CPU is located and waits for an idle CPU to execute the task. Then, a CPU with the lowest load is selected in the LLC domain, and it is determined whether the CPU can be used as a new selected CPU.If the conditions are met, the corresponding mapping table is updated. Test The test environment consists of machines with 384 CPUs and 4 NUMA nodes. The performance of Redis's set and get operations is tested, with a total of 384 instances, using the valkey_benchmark tool. ./target/release/scx_soft_domain -P redis-server 1.SET test metric cfs scx_soft_domain scx_soft_domain vs cfs rps 126557166.47 136862901.07 +8.14 % avg(ms) 1.497 1.392 p50(ms) 1.308 1.169 p95(ms) 2.664 2.446 p99(ms) 6.531 4.696 2.GET test metric cfs scx_soft_domain scx_soft_domain vs cfs rps 127227716.58 136673083.79 +7.42 % avg(ms) 1.486 1.391 p50(ms) 1.31 1.193 p95(ms) 2.523 2.363 p99(ms) 5.845 4.163 The current scheduler is specifically tuned for workloads that share the performance profile of Redis. In future updates, we plan to release additional optimisation policies for other workload types, such as MySQL‑like models.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scx_soft_domain: through topology-aware scheduling to improve performance
Background
In the performance test of multi-instance applications, such as Redis, a large number of CPU migrations across clusters or NUMA nodes are detected by capturing cputrace. In addition, there is cross-NUMA memory access. In addition, multiple Redis processes on the same CPU are switched back and forth. When the test pressure is high, the CPU competition is fierce.
We hope to reduce the Redis migration and competition and reduce the cross-NUMA memory access based on the topology information.
Introduce
scx_soft_domain is a scheduler dedicated to multi-instance services, such as Redis. It detects the CPU hardware topology (LLC and NUMA) and task affinity, and executes tasks on the CPUs in the domain where the tasks are located as much as possible. This reduces data transmission across LLCs and NUMAs, improves cache locality, and improves the overall system performance.
Core implementation logic
The CPU selection policy is implemented as follows: Multiple instances are evenly distributed on the machine. Within the same scheduling domain (that is, the same LLC or NUMA node), tasks are preferentially scheduled to CPUs in the domain, and cross-domain migration is avoided as much as possible.
When the load is high, load balancing is performed. First, whether the task can be directly executed on the selected CPU is quickly determined based on the load of the selected CPU and the running status of the task on the selected CPU. If the task cannot be directly executed on the selected CPU, the task is inserted into a user-defined queue of the LLC where the selected CPU is located and waits for an idle CPU to execute the task. Then, a CPU with the lowest load is selected in the LLC domain, and it is determined whether the CPU can be used as a new selected CPU. If the conditions are met, the corresponding mapping table is updated.
Test
The test environment consists of machines with 384 CPUs and 4 NUMA nodes. The performance of Redis's set and get operations is tested, with a total of 384 instances, using the valkey_benchmark tool. The kernel source code used in the test is obtained from https://atomgit.com/openeuler/kernel/tree/openEuler-26.09.
./target/release/scx_soft_domain -P redis-server
set
get
cargo build --release -p scx_soft_domainpassescargo fmt --checkpassesNotes
Currently, this feature only applies to the Redis performance improvement in the entire system scenario, including bare metal scenarios or container scenarios. In the future, more service scenarios will be supported.