Initial PR for scheduler scx_lunar

[WhitePeace36]

Hi,

I created a new cpu scheduler. Which i have been working on for quite some while now.

I think it works great and wanted to give other people also the opportunity to use it, so i am creating this PR.

How the scheduler works is in the README below.

To be honest i first created the loader in c++ and this is what i mostly used for development because i am not that into rust and i first tried to search if there would be any possibility that this repo would accept c++ loaders. But i don't think that would be a good idea, so i asked claude to translate the loader from c++ to rust and it works fine. So there looks to be no problem.
Only so that you are aware.

I created this PR before, but it was not up to par. I fixed up a few things, cleaned the commits and updated the readme.

Description:

Lunar

Introduction

Scx_lunar is a multipurpose scheduler which was originally invented with the goal to make frametimes in games as smooth as possible
But then it grew a little and changed to a desktop usage focused scheduler which focuses on IO bound threads.

Which makes the scheduler one of the best when it comes to responsiveness.

This scheduler uses only FIFO queues.

Explanation

The scheduler works with accounting of duty.

Duty goes from 0 to 1023.

The higher the duty number the more the task hogs cpu power.
The lower the more it is sleeping or dependent on io.

The duty is calculated from a window of the last 100ms

It is calculated like this.

duty = sleep_time * 1024 /(run_time + sleep_time + 1)

The Tier are calculated as Percent of the 1024 max duty value.

It has 5 tiers. Which are:

1. LC with duty <= 5%
2. INTERACTIVE with duty <= 20%
3. NORMAL with duty <= 40%
4. BATCH with duty <= 90%
5. GREEDY with duty <= 100%

All new tasks get thrown into greedy. And start with duty of 1023.
There is also a min. sample rate of the duty value to be eligible for promotion into higher tiers.

Each tier also has a slice time of 500us.

When a lower tier task is running at the moment a higher tier gets enqueued then the current task gets kicked and preempted.

MODES

This scheduler also has 2 modes.

--mode dsqs_per_llc

Where the above explained are available for each LLC. So more than one core pull from the same DSQs.

and:

--mode dsqs_per_cpu

DEFAULT MODE!
Where the above explained dsqs are available for each cpu core. So each core has its own queues.
This mode is used automatically when starting without start parameters.

Dispatch

For mode dsqs_per_cpu
Each core first tries to run its own queued tasks, then from another core from the same llc and then from core of other llcs.
From which core the core startes stealing is randomized for better load distribution.

for mode dsqs_per_llc
Each core tries to first to run from the dsqs of the llc from the core. Then it tries to steal from other llcs.

Testing

There where 2 design goals for this scheduler.

1. That music keeps playing normally when executing the cachyos benchmarker https://github.com/CachyOS/cachyos-benchmarker
2. To keep frametimes as smooth as possible with as little frametime spikes as possible.

As far as i have tested. Both modes do accomplish these tasks very well.

The only problem is i couldn't test the functionality with different llcs as i don't have such an cpu by hand.
The next thing is, that i mostly developed this scheduler with SMT disabled. As i found that SMT off works the best for this ryzen 5800x3d. But you can test both. Your mileage may vary.

[sirlucjan]

1. Please use the sched/experimental directory; it is intended for new projects.
2. Use cargo fmt after every time you edit a file

[sirlucjan]

I would also ask that you include some specific information about the scheduler in the PR description—how it works, what it’s used for, and what its characteristics are. A benchmark showing real-world results compared to EEVDF or other schedulers would also be welcome.

[wllclngn]

@WhitePeace36 Trust Piotr during this process. He's extremely knowledable of sched_ext and will try to assist as he can. That being said, more information on the scheduler itself, what it does, etc., would be appreciated in a PR. Feel free to view other accepted PRs for what is being looked for, etc. Best of luck!

[sirlucjan]

Since you're saying you're not very experienced with Rust, how do you see long-term maintenance of this scheduler? If users report bugs or future API changes require modifications, will you be comfortable maintaining the Rust implementation?

Thanks for running the CachyOS Benchmarker. Could you please include the benchmark results in the PR description? It would also be helpful to know the test environment (CPU, kernel version, workload, number of runs) so others can interpret or reproduce the results.

[sirlucjan]

The scheduler did not crash or cause the system to freeze under the load of the cachyos-benchmarker and compiling scx with cargo build --release --workspace—however, compared to EEVDF, the system was slightly less responsive.

Below are the results (EEVDF vs. scx_lunar)

[wllclngn]

For me, I had not extensively worked w/ Rust until this year. I think w/ your C/C++ knowledge you should be able to pick it up relatively quickly as per Piotr's question. Much of system-layer programming is headed in this direction, too. Scheduler shows a lot of potential.

[WhitePeace36]

@sirlucjan @wllclngn I mean it't not like i never did anything in rust. Around 1 year ago i got quite into it, but this was also the last time i touched it, so i forgot a lot along the way and because i am doing c++ every day, so that's also a reason. But i will just have to get back into it.

I will maintain it, because i also intend to use it myself :) And i am also already using it. :)

To be honest i never got the whole benchmark because i close to always just interrupted it. But right now i started it and let it complete for my ryzen 5800x3d but @sirlucjan it seems like you were faster than me. :)

[WhitePeace36]

@sirlucjan and yes, the scheduler quite heavily penalizes cpu greedy and spamming tasks and gives more time to shorter running tasks when possible.

@wllclngn thx again :) Yes, i think its something new i guess. ^^. At least on my side when testing mostly in different games it runs very well.

[sirlucjan]

@WhitePeace36
Don't take my questions as nitpicking. I'd rather ask questions now than make assumptions later.

Either way, I'm looking forward to the results. cachyos-benchmarker is a real stress test for schedulers and has exposed bugs or even caused crashes in the past. If your scheduler handled it without crashing, that's already a very good sign.

For me, stability is the most important thing. I also appreciate that the scheduler reports its version nicely in the logs, that kind of detail is important for us when debugging and supporting users.

[WhitePeace36]

@sirlucjan no worries. I don't take it as nitpick to be honest :) . This is nothing in comparison to stuff i have already experienced to be honest. I am more thankful to be honest :)

i got the benchmark results, from my pc now.

[WhitePeace36]

@sirlucjan do you maybe use SMT or have a cpu with more than one L3 cache ?

[sirlucjan]

My CPU: Ryzen 7 8845HS (8/16).

[WhitePeace36]

@sirlucjan hmm ok, so also 1 L3 cache. So i assume the difference is because of SMT being enabled. I would need to look into that i think, but maybe this will be also a downside of this design. I am not sure.

[WhitePeace36]

here is again a benchmark with dsqs_per_cpu instead of the default of dsqs_per_llc

[WhitePeace36]

i found a issue which i overlooked after the lastest rework. But now it is fixed. Now dsqs_per_cpu seems to be better for performance but i assume that idle power consumption will be higher than with dsqs_per_llc.

[arighi]

I think I mentioned this already, could you please write a proper commit description for each commit? Take a look at how other commits are written and follow the same style.

[WhitePeace36]

@arighi ahh sry, it seems like i misunderstood what you meant. I will update them later, when i am at home.

[WhitePeace36]

@arighi i hope this is ok so. I am not sure what else to write to be honest.

[arighi]

@arighi i hope this is ok so. I am not sure what else to write to be honest.

It's better, some comments:

• each commit should represent a coherent logical change. Avoid commits whose only purpose is to fix issues introduced by the immediately preceding commit in the same PR. Instead, fold those fixes into the original commit so that every commit stands on its own.
• In the initial commit briefly explain what the scheduler does, what are the benefits, try motivate why we should merge it (just a brief overview without going too deep into details, then you can explain more details in the documentation)
• Add a Signed-off-by line (see other commits)

Thanks.

[WhitePeace36]

The scheduler did not crash or cause the system to freeze under the load of the cachyos-benchmarker and compiling scx with cargo build --release --workspace—however, compared to EEVDF, the system was slightly less responsive.

Below are the results (EEVDF vs. scx_lunar)

I have done some further testing with SMT on and found the issue reproducible. The issue seems to be the lock contention when there is only 1 amount of the dsqs per L3 domain. The more cores in one L3 domain, the more the lock contention.

benchmarks with ryzen 5800x3d SMT ON and dsqs_per_lcc in comparison to eevdf.

here is the other benchmark with also ryzen 5800x3d SMT ON but with dsqs_per_cpu.

So based on feelings and benchmarks i made the dsqs_per_cpu the default mode.

[sirlucjan]

[WhitePeace36]

@wllclngn looks good to me :) seems to be exactly how its designed.

Btw i just pushed again because i found a not optimal #define value for the spammer detection.

#define TASK_LAST_SPAWN_THRESH 70000000ULL

that is all i changed.

With this the spammer detection works a lot better.

[WhitePeace36]

the P99 latency should now be quite a bit better and the scheduler is feeling a lot smoother.

[WhitePeace36]

i just updated the scheduler. I found the issue yesterday which i have been searching for for quite a while.

these are the new test results.
It is now even more responsive under load when running the cachyos benchmarks than eevdf except for the "perf sched msg fork thread" test. There it seems to be a less responsive.

[WhitePeace36]

@sirlucjan i made quite a few changes now in the scheduler.

I have changed also the PR description.

It now uses a thing called duty. which goes from 0-1023 for the classifications and also uses preemption.

This fixed some strange issues and makes the scheduler better overall.