Skip to content

cli: add --default-timeout option for tests with no explicit timeout#596

Merged
MrAnno merged 1 commit into
Snaipe:bleedingfrom
PoQuatre:bleeding
May 25, 2026
Merged

cli: add --default-timeout option for tests with no explicit timeout#596
MrAnno merged 1 commit into
Snaipe:bleedingfrom
PoQuatre:bleeding

Conversation

@PoQuatre

Copy link
Copy Markdown
Contributor

This PR introduces a new --default-timeout CLI option (and its corresponding criterion_options.default_timeout field) that applies a fallback timeout to any test that has no timeout set, neither directly on the test nor inherited from its suite.

The existing --timeout option acts as a hard cap: it overrides every test's timeout downward if the test's own timeout exceeds the cap. --default-timeout fills the complementary gap: it only kicks in when no timeout would otherwise apply, leaving explicitly-timed tests completely untouched.

This would also somewhat resolve #279

Timeout resolution order (after this PR)

  1. Suite timeout (if set)
  2. Per-test .timeout (overrides suite)
  3. --default-timeout applied only when the result of steps 1–2 is <= 0
  4. --timeout cap applied only when the result so far exceeds the cap

Code style notes

Uncrustify was run over the touched files. Two issues came up:

  • Pre-existing errors in untouched files. Uncrustify reported style violations in files not modified by this PR. Those were left alone to keep the diff focused and avoid muddying the review with unrelated formatting churn.

  • Backslash alignment in the USAGE macro (src/entry/params.c). Uncrustify suggested de-aligning the continuation backslashes added for the new --default-timeout line. The rest of the macro uses aligned backslashes as an intentional style choice, so the new lines were written to match the surrounding code rather than follow the tool's suggestion.

Open questions / uncertainties

Location of the new tests

The cram test (test/cram/timeout.t) references default-timeout.c.bin by name, which is only built when the full test suite is configured. It is unclear whether this file belongs under test/cram/ (which typically contains self-contained script tests) or whether it would be better placed alongside the full-test fixtures it depends on, or driven from test/full/meson.build directly.

Help message clarity for --timeout vs --default-timeout

The current help strings are:

--timeout [TIMEOUT]: set a timeout (in seconds) for all tests
--default-timeout [TIMEOUT]: set a default timeout (in seconds) for tests with no explicit timeout

These may not make the semantic difference obvious enough to end users, since "for all tests" could be read as "applies universally" without conveying that it acts as a cap. A clearer phrasing might be something like:

--timeout [TIMEOUT]: cap the timeout (in seconds) for all tests, overriding any per-test value that exceeds it
--default-timeout [TIMEOUT]: set a fallback timeout (in seconds) applied only to tests with no timeout set

@MrAnno MrAnno left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR and the meticulous work :)

Timeout resolution order

It sounds perfect, it's also backward-compatible.

Code style notes

Thanks again for your precision, it looks perfect to me.
Yeah, the format Uncrustify wants to produce is far from what we can see on bleeding.

Location of the new tests

Timing-based tests aren't the most warmly welcomed creatures in test suites.

sample / timeout.c is already in the test suite, but it doesn't validate much, it's more like a smoke test.

I think the location of default-timeout.c is right. What's worth considering is how many times we want to run these tests and for what purpose.
default-timeout currently runs three times: once as part of the "full" test suite, and twice again as cram-validated tests.

I think it would be worth not registering default-timeout as a "full" test and letting cram validate it instead. Or if you prefer keeping it in both suites, we can remove the timeout sample from the "full" test suite instead (running 2 --always-succeed smoke tests around timeouts is not really useful).

Help message clarity for --timeout vs --default-timeout

I like your suggestion, it clarifies everything discussed in #279. Would you like to add it to this PR?

@PoQuatre

Copy link
Copy Markdown
Contributor Author

Thank you for the review and the kind words!

I think it would be worth not registering default-timeout as a "full" test and letting cram validate it instead.

I'll go with this option, removing the test() call for default-timeout from test/full/meson.build and letting the cram tests be the sole validators. The binary still needs to be built (the cram suite references it by name), so I'll keep the executable() definition and just drop the test() entry. That leaves sample / timeout.c as the one --always-succeed smoke test for timeouts, and timeout.t as the real validation.

Would you like to add it to this PR?

Yes, happy to include the updated help strings in this PR. I'll update both --timeout and --default-timeout to the phrasing from the PR description.

Comment thread src/core/runner_coroutine.c Outdated
@PoQuatre

Copy link
Copy Markdown
Contributor Author

I just made the change you suggested, and switched the order in which the --timeout and --default-timeout are checked and applied.

@MrAnno MrAnno left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@MrAnno MrAnno merged commit c986940 into Snaipe:bleeding May 25, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Criterion hangs - cannot define global max timeout (2.3.3 & 2.3.2)

2 participants