Skip to content

scx_layered: skip absent CPUs when initializing per-cpu ctxs - #3718

Open
hodgesds wants to merge 1 commit into
sched-ext:mainfrom
hodgesds:layered-nr-cpus-fix
Open

scx_layered: skip absent CPUs when initializing per-cpu ctxs#3718
hodgesds wants to merge 1 commit into
sched-ext:mainfrom
hodgesds:layered-nr-cpus-fix

Conversation

@hodgesds

Copy link
Copy Markdown
Contributor

init_cpus() iterated 0..NR_CPUS_POSSIBLE and unwrapped topo.all_cpus.get(&cpu), which panics for possible-but-not-present CPUs. On bare metal possible == online so this never triggers, but a VM booted on a kernel with CONFIG_NR_CPUS > vCPU count (e.g. an fbk kernel with NR_CPUS=512 booted with 384 vCPUs) has a sparse possible range and hits the panic at the first missing slot.

Guard the topo lookup and skip absent slots. The percpu map still gets one entry per possible CPU (the ctx stays zero-initialized for absent slots, which is fine because BPF never runs on them), so downstream code that indexes cpu_ctxs[cpu_id] keeps working.

Validated with virtme-run using --cpus 384 --qemu-opts -smp cpus=384,maxcpus=512 (KVM capped online at 256, so the actual guest saw possible=0-511 online=0-255):

Before (panic):

VM_UP
POSSIBLE=0-511
ONLINE=0-255
INFO scx_layered: CPUs: online/possible=256/512 nr_cores=256
INFO scx_layered: Running scx_layered (build ID: 1.1.2 x86_64-unknown-linux-gnu/debug)
thread 'main' (3014) panicked at scheds/rust/scx_layered/src/main.rs:2428:52:
called Option::unwrap() on a None value
EXITCODE=101

After (clean startup and shutdown):

VM_UP
POSSIBLE=0-511
ONLINE=0-255
INFO scx_layered: CPUs: online/possible=256/512 nr_cores=256
INFO scx_layered: Running scx_layered (build ID: 1.1.2 x86_64-unknown-linux-gnu/debug)
INFO scx_layered: Layered Scheduler Attached. Run scx_layered --monitor for metrics.
EXIT: unregistered from user space
INFO scx_layered: Unregister scx_layered scheduler
EXITCODE=0

init_cpus() iterated 0..NR_CPUS_POSSIBLE and unwrapped
topo.all_cpus.get(&cpu), which panics for possible-but-not-present
CPUs. On bare metal possible == online so this never triggers, but
a VM booted on a kernel with CONFIG_NR_CPUS > vCPU count (e.g. an
fbk kernel with NR_CPUS=512 booted with 384 vCPUs) has a sparse
possible range and hits the panic at the first missing slot.

Guard the topo lookup and skip absent slots. The percpu map still
gets one entry per possible CPU (the ctx stays zero-initialized
for absent slots, which is fine because BPF never runs on them),
so downstream code that indexes cpu_ctxs[cpu_id] keeps working.

Validated with virtme-run using --cpus 384 --qemu-opts -smp
cpus=384,maxcpus=512 (KVM capped online at 256, so the actual
guest saw possible=0-511 online=0-255):

Before (panic):

  VM_UP
  POSSIBLE=0-511
  ONLINE=0-255
  INFO scx_layered: CPUs: online/possible=256/512 nr_cores=256
  INFO scx_layered: Running scx_layered (build ID: 1.1.2 x86_64-unknown-linux-gnu/debug)
  thread 'main' (3014) panicked at scheds/rust/scx_layered/src/main.rs:2428:52:
  called `Option::unwrap()` on a `None` value
  EXITCODE=101

After (clean startup and shutdown):

  VM_UP
  POSSIBLE=0-511
  ONLINE=0-255
  INFO scx_layered: CPUs: online/possible=256/512 nr_cores=256
  INFO scx_layered: Running scx_layered (build ID: 1.1.2 x86_64-unknown-linux-gnu/debug)
  INFO scx_layered: Layered Scheduler Attached. Run `scx_layered --monitor` for metrics.
  EXIT: unregistered from user space
  INFO scx_layered: Unregister scx_layered scheduler
  EXITCODE=0
@likewhatevs

Copy link
Copy Markdown
Contributor

I think that's a bug in the VMM if it happens right? Mentioning this because I addressed this in ktstr a week or two ago while productionizing things to enable use here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants