Stream verifier scheduler output, quiet VMM diagnostics, fix summary glyph - #26
Merged
Conversation
A failed scheduler cell in CI buried its actual failure under hundreds of host-side diagnostic lines: one no_perf_mode affinity line per vCPU, the BSP run-loop trace, and the CLEANUP teardown timings. None of them help someone debugging their scheduler — they exist for debugging ktstr itself. Route them through debug_logging_enabled(), on when KTSTR_DEBUG=1 or RUNNER_DEBUG=1 — the latter is what GitHub Actions exports on "re-run with debug logging", so a failed CI job re-run in debug mode gets the full diagnostics without a ktstr-specific knob. Warnings, watchdog-deadline diagnostics, VM-setup timings, and the interactive shell's abnormal-exit line stay unconditional.
U+1F1FD REGIONAL INDICATOR SYMBOL LETTER X is half of a flag pair, not a standalone emoji: alone it renders as a boxed X, a blue X, or tofu depending on the font stack, and its ambiguous width misaligns the summary-grid columns. ❎ (U+274E) is a standard emoji with the same width as ✅/❌, so the mixed cell renders everywhere the other two states already do.
A verifier cell that hung after attach reported only "scheduler: UNKNOWN — VM timed out before exit": the scheduler's output was captured into a merged /tmp/sched.log inside the guest and shipped to the host only by the teardown dump, which a watchdog kill never reaches. Even on runs that did dump, the formatter printed only the libbpf verifier-log region extracted from the blob — dropping the scheduler's own messages around it — and stderr was merged indistinguishably into stdout, never reaching the test's stderr. Wire the scheduler child's stdout and stderr through per-stream forwarder threads that tee each pipe read into the merged log file (keeping every existing dump path intact) and ship it immediately as new SCHED_STDOUT / SCHED_STDERR bulk frames. The host records both streams unconditionally, so whatever arrived before a watchdog kill still renders. The stdout section now shows the full scheduler stdout with the verifier trace collapsed in place instead of extract-and-drop, and the captured stderr is emitted to the test's real stderr (skipping the merged-dump fallback when a live stream arrived, so the two sections never duplicate). The dump paths wait (bounded) for the forwarders to drain the dead child's pipes so the merged file stays as complete as when the child wrote it directly.
New minor: verifier cells stream the scheduler's stdout/stderr live from the guest — output survives watchdog timeouts, stdout renders in full with the verifier trace collapsed in place, and stderr reaches the test's stderr; noisy host-side VMM diagnostics (no_perf_mode masks, BSP run-loop trace, CLEANUP timings) are gated behind KTSTR_DEBUG=1 / RUNNER_DEBUG=1; the mixed verifier-summary cell renders ❎ instead of the font-dependent 🇽.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes three CI-log ergonomics problems surfaced by the scx verifier runs (sched-ext/scx#3700):
Scheduler stdout/stderr now reach the test's stdout/stderr — including on timeout
The scheduler's output was captured into a merged
/tmp/sched.login the guest and shipped only by the teardown dump, which a watchdog kill never reaches — a hung cell reported onlyscheduler: UNKNOWN — VM timed out before exitwith no scheduler output at all. Even on dumping runs, the formatter printed only the extracted libbpf verifier-log region (dropping the scheduler's own messages) and stderr never reached the test's stderr.Now the scheduler child's stdout/stderr are piped through per-stream forwarder threads that tee into the merged log file (all existing dump paths intact) and stream live over new
SCHED_STDOUT/SCHED_STDERRbulk frames. Whatever arrived before a watchdog kill still renders. Stdout renders in full with the verifier trace collapsed in place; stderr is emitted to the test's real stderr. The dump paths wait (bounded) for the forwarders to drain so the merged file stays as complete as when the child wrote it directly.VMM diagnostics gated behind debug logging
The per-vCPU
no_perf_modemask lines,BSP:run-loop trace, andCLEANUP:teardown timings buried scheduler failures in CI logs. They now print only whenKTSTR_DEBUG=1orRUNNER_DEBUG=1(the variable GitHub Actions exports on "re-run with debug logging"). Warnings, watchdog-deadline diagnostics, and VM-setup timings stay unconditional.Verifier summary glyph
The mixed-cell 🇽 (a lone regional-indicator codepoint) renders as a boxed/blue X or tofu depending on the font stack and misaligns the grid; it's now ❎, same width as ✅/❌.
Test plan
cargo nextest run --no-fail-fast— 10382 passed, 0 failed (includes the VM-bootingverifier_pipelinecells; the previously-flakyverifier_cycle_collapsepassed 8/8 consecutive isolated runs after the forwarder-drain fix)just lint— all legs green (fmt, clippy, clippy wprof,integration, doc-strict, docsrs-mode)just test-doc,just test-macros— greenscx_cosmosfrom add tests against distro kernels sched-ext/scx#3700 with ktstr path-patched, rancargo ktstr verifier --scheduler cosmos --kernel ubuntu; failed cells now show the scheduler's real error (Error: Failed to load BPF program — Invalid argument) on stdout, scheduler stderr in the stderr section, no duplicated sections, and none of the gated diagnostic noise