Skip to content

Add per-test configuration overrides#770

Open
MatthewMckee4 wants to merge 1 commit into
mainfrom
worktree-nested-wandering-acorn
Open

Add per-test configuration overrides#770
MatthewMckee4 wants to merge 1 commit into
mainfrom
worktree-nested-wandering-acorn

Conversation

@MatthewMckee4
Copy link
Copy Markdown
Owner

@MatthewMckee4 MatthewMckee4 commented May 6, 2026

Summary

Adds [[profile.<name>.overrides]] entries that pair a filter expression with one or more option fields. The first matching override wins for each test, falling back to the profile-level value when nothing matches.

[profile.default.test]
retry = 1
timeout = 30.0

[[profile.default.overrides]]
filter = "tag(network)"
retries = 5
timeout = 300.0

[[profile.default.overrides]]
filter = "tag(unit)"
retries = 0
slow-timeout = 0.5

The override entry currently supports retries, timeout, and slow-timeout, mirroring the equivalent profile-level fields. A matching override with a non-positive timeout or slow-timeout disables the corresponding limit for the matched test even when the profile sets one.

Filters use the existing filterset DSL (tag(...), test(...), and boolean combinators) and are validated at TOML parse time via a ValidatedFilter newtype, so the worker never re-parses or panics. Overrides defined under [profile.default] are layered beneath named-profile entries: a [[profile.ci.overrides]] block takes precedence when the ci profile is active, with default-profile entries as the fallback. Overrides are forwarded from the main process to workers via a hidden, JSON-encoded --override-json flag.

Closes #555. Closes #577.

Test Plan

ci

@MatthewMckee4 MatthewMckee4 added configuration Related to configuring Karva retries Related to retrying and flaky test handling labels May 6, 2026
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 6, 2026

Merging this PR will improve performance by 10.06%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime karva_benchmark 56.9 s 51.7 s +10.06%

Comparing worktree-nested-wandering-acorn (40c3e12) with main (eb3c800)

Open in CodSpeed

Introduces `[[profile.<name>.overrides]]` entries that pair a filter
expression with one or more option fields. The first matching override
wins, falling back to the profile-level value when no override matches.

```toml
[profile.default.test]
retry = 1
timeout = 30.0

[[profile.default.overrides]]
filter = "tag(network)"
retries = 5
timeout = 300.0

[[profile.default.overrides]]
filter = "tag(unit)"
retries = 0
slow-timeout = 0.5
```

The override entry currently supports `retries`, `timeout`, and
`slow-timeout`, mirroring the equivalent profile-level fields. A
matching override with a non-positive `timeout` or `slow-timeout`
disables the corresponding limit for the matched test even when the
profile sets one.

Filters use the existing filterset DSL (`tag(...)`, `test(...)`, and
boolean combinators) and are validated at TOML parse time via a
`ValidatedFilter` newtype, so the worker never re-parses or panics.
Overrides defined under `[profile.default]` are layered beneath named-
profile entries: a `[[profile.ci.overrides]]` block takes precedence
when the `ci` profile is active, with default-profile entries as the
fallback.

Closes #555. Closes #577.
@MatthewMckee4 MatthewMckee4 force-pushed the worktree-nested-wandering-acorn branch from 35ee942 to 40c3e12 Compare May 6, 2026 07:33
@MatthewMckee4 MatthewMckee4 added the test-running Related to running tests with karva label May 6, 2026
@MatthewMckee4 MatthewMckee4 changed the title Add per-test retry overrides Add per-test configuration overrides May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

configuration Related to configuring Karva retries Related to retrying and flaky test handling test-running Related to running tests with karva

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-test overrides in configuration Per-test retry overrides

1 participant