Skip to content

Add #[ktstr_scenario]: a test's workload as an extractable value - #43

Closed
rrnewton wants to merge 1 commit into
likewhatevs:mainfrom
rrnewton:feat/ktstr-scenario-dsl
Closed

Add #[ktstr_scenario]: a test's workload as an extractable value#43
rrnewton wants to merge 1 commit into
likewhatevs:mainfrom
rrnewton:feat/ktstr-scenario-dsl

Conversation

@rrnewton

Copy link
Copy Markdown

What and why

A ktstr test's workload is arbitrary Rust that happens to end in
execute_steps(ctx, steps). The shape of most of those bodies is declarative
— a list of steps — but nothing in the type system says so, and nothing can
recover the step list without booting a guest and running the test. That is a
shame for anything that wants to read a workload rather than execute it.

This adds ScenarioDef (the step list, plus the optional Assert override
execute_steps_with takes, lifted out of the body into a value) and
#[ktstr_scenario], the restricted entrypoint that consumes one. Five existing
tests are ported to it.

// before
#[ktstr_test(scheduler = KTSTR_SCHED, llcs = 1, cores = 2, threads = 1, ...)]
fn sched_basic_proportional(ctx: &Ctx) -> Result<AssertResult> {
    let steps = vec![Step {
        setup: vec![ctx.cgroup_def("cg_0"), ctx.cgroup_def("cg_1")].into(),
        ops: vec![],
        hold: HoldSpec::FULL,
    }];
    execute_steps(ctx, steps)
}

// after
#[ktstr_scenario(scheduler = KTSTR_SCHED, llcs = 1, cores = 2, threads = 1, ...)]
fn sched_basic_proportional() -> ScenarioDef {
    ScenarioDef::with_defs(vec![CgroupDef::named("cg_0"), CgroupDef::named("cg_1")])
}

It does not fork #[ktstr_test]

That is the main design constraint. #[ktstr_scenario] emits three items and
hands the third to ktstr_test_impl verbatim:

  1. the author's function, renamed __ktstr_scenario_body_<name>, body and
    attributes untouched;
  2. __ktstr_scenario_def_<name>() -> ScenarioDef, coercing (1)'s return value
    via Into (so a body may return ScenarioDef, Vec<Step> or a bare
    Step), plus a ScenarioEntry registration in the new KTSTR_SCENARIOS
    slice;
  3. fn <name>(ctx: &Ctx) -> Result<AssertResult> { <def>().run(ctx) }, passed
    to ktstr_test_impl with the attribute list unchanged.

So attribute parsing, every cross-attribute validation, and every existing
registration (KTSTR_TESTS, the manifest stamp, both admission stamps, the
#[test] wrapper) are literally the same code — any future #[ktstr_test]
field is inherited by construction. ScenarioDef::run dispatches to
execute_steps_with, so there is no second execution engine either.

delegates_to_ktstr_test_verbatim pins exactly that: the scenario expansion
must contain ktstr_test_impl's own output for the synthesized runner, token
for token
, over a broad attribute set. That is the equivalence evidence — no
field-by-field list to keep in sync, and an attempt to intercept or rewrite the
delegation breaks the test immediately.

What is restricted is the function, not the attribute grammar

  • No arguments — in particular no ctx: &Ctx; that is what makes the body
    host-buildable.
  • Not async, not generic, no where clause — the generated builder is a
    plain fn() -> ScenarioDef.
  • post_vm / post_vm_unconditional rejected — a host-side callback is
    the arbitrary Rust this entrypoint exists to exclude. Use #[ktstr_test].

Each rejection names its reason and points at #[ktstr_test]; two are pinned
as compile_fail fixtures.

The &Ctx restriction costs less than it looks

Nearly every &Ctx use in an existing scenario body is ctx.cgroup_def(n),
which is CgroupDef::named(n).workers(ctx.workers_per_cgroup). A CgroupDef
that leaves its worker count unset resolves through the same default anyway:
empty worksWorkSpec::default()num_workers: None
resolve_num_workers's unwrap_or(ctx.workers_per_cgroup). The two spell the
same workload; the ctx-free one just leaves it symbolic for the runner to bind,
which is the more useful form for a value meant to describe a workload rather
than one host's instance of it.

Honest boundary

Setup::Factory(fn(&Ctx) -> Vec<CgroupDef>) remains the genuine escape hatch,
and a ScenarioDef may legally hold one. Rather than pretend otherwise,
is_declarative() reports it, so a consumer reading scenarios as data skips
such a scenario deliberately instead of silently believing an empty cgroup
list. every_registered_scenario_is_declarative asserts it over the whole
registry.

Naming

ktstr_scenario is new public surface and the name is open to review. I kept
it over the alternatives (ktstr_workload, declarative_test, scenario_test)
because it pairs with the ScenarioDef / ScenarioEntry / KTSTR_SCENARIOS
vocabulary and reads as a sibling of ktstr_test. Its one weakness is that it
does not say "test" — happy to rename; it is two non-derived identifiers plus
re-blessing two .stderr snapshots.

Test plan

  • 7 ScenarioDef unit tests (src/scenario/def/tests.rs).
  • 7 macro-expansion tests (ktstr-macros), including the verbatim-delegation
    pin and the rejection diagnostics.
  • 3 host-side extraction tests over the ported set, checking the extracted
    values against independently written expectations — cgroup names and order,
    disjoint cpuset indices, per-step 0.5 holds, and sched_perf_positive's
    min_iteration_rate / max_gap_ms surviving into the extracted checks.
  • 2 compile_fail fixtures; the full trybuild suite passes with no existing
    snapshot disturbed.

Ran green locally: cargo check and cargo clippy on both feature sets,
cargo fmt --check, rustdoc-warnings-as-errors, the docs.rs-mode check,
just compile-fail, just devdep-isolation.

Not run: the KVM integration suite. My host cannot run it — ktstr's content
cache hard-requires a reflink (cache/content.rs FICLONE, no EXDEV
fallback), and here /usr/bin/false (on /) and ~/.cache/ktstr (on
/home) are separate btrfs filesystems, so scheduler-artifact publication
fails with Invalid cross-device link before any test runs. I confirmed this
is not caused by this branch by running the identical command on an untouched
checkout of the base commit — it fails byte-identically. CI's KVM runners
should exercise the five ported tests normally. (just link-check likewise
only fails here for a missing mdbook; this diff touches no guide content.)

Note on an incidental comment fix

tests/ktstr_sched_tests.rs carried a note claiming plain #[test] functions
in a KtstrTestEntry-registering binary are invisible to the runner.
list_plain_tests re-emits them; verified against that binary's own
NEXTEST=1 --list output. Corrected, and the new extraction tests are plain
#[test]s in that file that do run.

A ktstr test's workload is arbitrary Rust that happens to end in
`execute_steps(ctx, steps)`. The SHAPE of most of those bodies is
declarative — a list of steps — but nothing in the type system says
so, and nothing can recover the step list without booting a guest and
running the test. That is a shame for anything that wants to READ a
workload rather than execute it.

Introduce `ScenarioDef`: the step list (plus the optional `Assert`
override `execute_steps_with` takes) lifted out of the body into a
value that can be built, inspected and printed on the host with no VM
and no `&Ctx`. `ScenarioDef::run` dispatches to `execute_steps_with`,
so there is no second execution engine — a scenario runs down exactly
the path a hand-written body runs down.

Add `#[ktstr_scenario]` as the entrypoint that consumes one. It does
NOT fork `#[ktstr_test]`: it emits the author's function renamed, a
canonical `fn() -> ScenarioDef` builder that coerces the return value
via `Into` (so a body may return `ScenarioDef`, `Vec<Step>` or a bare
`Step`), a `ScenarioEntry` registration in the new `KTSTR_SCENARIOS`
slice, and a synthesized `fn <name>(ctx: &Ctx) -> Result<AssertResult>`
that it hands to `ktstr_test_impl` unchanged. Attribute parsing,
cross-attribute validation and every existing registration are
therefore literally the same code, and any future `#[ktstr_test]`
field is inherited by construction. `delegates_to_ktstr_test_verbatim`
pins that: the scenario expansion must CONTAIN `ktstr_test_impl`'s own
output for the runner, token for token.

The function is what is restricted, not the attribute grammar. No
parameters — in particular no `&Ctx`, which is what makes the body
host-buildable; no `async`, generics or `where` clause, which the
generated builder could not call; and no `post_vm` /
`post_vm_unconditional`, since a host-side callback is the arbitrary
Rust this entrypoint exists to exclude. Each rejection names its
reason and points at `#[ktstr_test]`.

The `&Ctx` restriction costs less than it looks. Nearly every `&Ctx`
use in an existing scenario body is `ctx.cgroup_def(n)`, which is
`CgroupDef::named(n).workers(ctx.workers_per_cgroup)` — and a
`CgroupDef` that leaves its worker count unset resolves through the
same default anyway (empty `works` -> `WorkSpec::default()` ->
`num_workers: None` -> `resolve_num_workers`'s
`unwrap_or(ctx.workers_per_cgroup)`). The two spell the same workload;
the ctx-free one just leaves it symbolic for the runner to bind, which
is the more useful form for a value that is meant to describe a
workload rather than one host's instance of it.

`Setup::Factory(fn(&Ctx) -> Vec<CgroupDef>)` remains the genuine
escape hatch, and a `ScenarioDef` may legally hold one. Rather than
pretend otherwise, `is_declarative()` reports it, so a consumer reading
scenarios as data skips such a scenario deliberately instead of
silently believing an empty cgroup list.

Port the five pure-scenario tests in `tests/ktstr_sched_tests.rs`
(`sched_basic_proportional`, `sched_cpuset_split`, `sched_dynamic_add`,
`sched_verifier_stats_populated`, `sched_perf_positive` — the last
exercising the checks path). Their attributes, topology, gates and
runtime behaviour are unchanged; the bodies lose 54 lines of `Step {
setup: .., ops: vec![], hold: .. }` boilerplate.

Also correct a stale note in that file claiming plain `#[test]`
functions in a `KtstrTestEntry`-registering binary are invisible to the
runner. `list_plain_tests` re-emits them; verified against this
binary's own `NEXTEST=1 --list` output, and the new
scenario-extraction tests at the end of the file are plain `#[test]`s
that do run.

Test: 7 `ScenarioDef` unit tests, 7 macro-expansion tests (incl. the
verbatim-delegation pin), 3 host-side extraction tests over the ported
set, 2 compile-fail fixtures. fmt / clippy (both feature sets) /
doc-strict / docs.rs-mode all clean; the full compile_fail suite passes
with no existing snapshot disturbed.
@rrnewton

Copy link
Copy Markdown
Author

Superseded by #47, which contains this work plus the rest of the series in one branch, rebuilt on current main. Closing to keep one review surface. The branch behind this PR is left intact.

@rrnewton rrnewton closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant