LTP: support re-running individual ltp tests with retry entries by Avinesh · Pull Request #27251 · os-autoinst/os-autoinst-distri-opensuse

Avinesh · 2026-11-02T22:35:00Z

Some LTP tests sporadically fail for known reasons and pass when run again, we do not add them in known issues to softfail, so that we do not hide any bugs. So the only remedy is to restart the job (several times), which reruns every test/module in that job and that cost machine time and delays QA process unnecessarily.

This provides an option to add retry: N entries for such issues to the known issues file. If a failed test has a matching entry, only that test is run again in the same module, up to N more times.

A run passes: the module is a softfail. The failed runs stay visible as steps (..._retry<N>).
No run passes: the module fails, as before.
Tests without a retry entry are not affected.

    test:
    - retry: N
      retval: ^rc$
      log: <fail log pattern>
      message: unstable test

retval and the new log selector (regex on the test fail log) limit the retry to the known failure.

A retry entry does not accept the failure as a known issue. find_whitelist_entry() skips it, so a known issue for the same test needs its own entry. Kernel errors on the serial console are checked once, after the last run, so a kernel error from a failed run still fails the module.

Verification runs:

Baseline example: https://openqa.suse.de/tests/24866419 (check the failed tests and follow the cloned chain)

with this PR:

ksm0_* :
https://openqa.suse.de/tests/25314604#step/ksm04/16
https://openqa.suse.de/tests/25314596#step/ksm02/16
https://openqa.suse.de/tests/25314598#step/ksm04_1/16

with retry entry

    ksm04:
    - arch: ppc64le$
      retval: ^1$
      log: 'TFAIL: pages_sharing is not'
      retry: 4
      message: Random timeouts or small differences in one or two KSM stats

memcg_ :

https://openqa.suse.de/tests/25317929#step/memcg_stat_rss/16
https://openqa.suse.de/tests/25317937#step/memcg_stat_rss/24
https://openqa.suse.de/tests/25317932#step/memcg_stat/16
https://openqa.suse.de/tests/25317941#step/memcg_subgroup_charge/16
https://openqa.suse.de/tests/25317933#step/memcg_move_charge_at_immigrate/24 (retries getting exhausted)

with retry entry

     memcg_stat:
     - arch: ppc64le$
       retval: ^(2|6)$
       log: 'TBROK: timed out on memory.usage_in_bytes'
       retry: 3 
       message: timeout issue

pids_9_ :
https://openqa.suse.de/tests/25255329#step/pids_9_100/16
https://openqa.suse.de/tests/25255331#step/pids_9_50/16
https://openqa.suse.de/tests/25255350#step/pids_9_10/16

    pids_9_50:
    - product: sle:(12-SP5|15-SP[45])$
      retval: ^4$
      log: 'TBROK: unlinkat\(.*\): EBUSY'
      retry: 5
      message: sometimes fail with unlink():EBUSY

An entry with 'retry: N' only marks a flaky test which should run again. It does not accept the failure as a known issue. Add find_retry_entry() to look it up, and make find_whitelist_entry() skip such entries. Both share the key walk, now in _find_entries().

pevik · 2026-11-03T10:08:10Z

+        $result_export = $self->run_attempt($tinfo);
+        last unless ($self->{result} // '') eq 'fail';
+
+        $entry = LTP::WhiteList->new()->find_retry_entry($self->{ltp_env}, $runfile, $test->{name});


very nit: I'm always a bit scared of while (1), I'd prefer to have for loop, which somehow evaluates the conditions.

very nit: I'm always a bit scared of while (1), I'd prefer to have for loop, which somehow evaluates the conditions.

I can't think of any better way to avoid while (1) and I think it is fine with definite exit checks -
the run didn't fail, or no retries are left.

pevik

LGTM, thank you! If we agree it'd be great to have it also in generic test runner (for xfstests at least).
@mdoucha Can you please review?

If the known issues have a 'retry: N' entry for a failed test, run it again in the same module, up to N more times. A passing run makes the module a softfail, otherwise it fails as before. 'retval' and 'log' regex limits the retry to a known failure message. run() now loops over run_attempt(), which holds its old body.

pevik · 2026-11-03T11:18:56Z

Something related to restarts (not to known issues): reconnect_mgmt_console sporadic failures would be great if they get restarted automatically.
https://openqa.opensuse.org/tests/6313292#step/reconnect_mgmt_console/1

czerw · 2026-11-03T15:40:34Z

Something related to restarts (not to known issues): reconnect_mgmt_console sporadic failures would be great if they get restarted automatically. https://openqa.opensuse.org/tests/6313292#step/reconnect_mgmt_console/1

openQA can restart conditionally based on result, but I didn't try it.

czerw

Two prose fixes: the second commit message says 'retval' and 'log' regex limits the retry - s/regex limits/regexes limit/; it stays in git history, so it needs an amend. In this description: s/cost machine time/costs machine time/.

czerw · 2026-11-03T16:46:18Z

+
+=cut
+
+sub find_retry_entry {


No unit test covers find_retry_entry or the skip of retry entries in find_whitelist_entry, so a regression there would pass CI silently. Add cases to t/07_ltp_whitelist.t.

rbmarliere

Kselftests VR https://openqa.suse.de/tests/25347940 -- no regression

Overall LGTM!

Avinesh requested review from SeroSun, coolgw, czerw, frankenmichl, lansuse, mdoucha, pevik, rbmarliere and schlad as code owners November 2, 2026 22:35

pevik reviewed Nov 3, 2026

View reviewed changes

Comment thread tests/kernel/run_ltp.pm Resolved

pevik reviewed Nov 3, 2026

View reviewed changes

pevik previously approved these changes Nov 3, 2026

View reviewed changes

Avinesh dismissed pevik’s stale review via 5692c22 November 3, 2026 11:09

Avinesh force-pushed the module-rerun branch from 91d8d7b to 5692c22 Compare November 3, 2026 11:09

czerw reviewed Nov 3, 2026

View reviewed changes

rbmarliere approved these changes Nov 4, 2026

View reviewed changes

Navigation Menu

Repository navigation

LTP: support re-running individual ltp tests with retry entries - #27251

LTP: support re-running individual ltp tests with retry entries#27251
Avinesh wants to merge 2 commits into
os-autoinst:masterfrom
Avinesh:module-rerun

Avinesh commented Nov 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

pevik Nov 3, 2026

Uh oh!

Avinesh Nov 3, 2026

Uh oh!

pevik left a comment •

edited

Loading

Uh oh!

pevik commented Nov 3, 2026

Uh oh!

czerw commented Nov 3, 2026

Uh oh!

czerw left a comment

Uh oh!

czerw Nov 3, 2026

Uh oh!

rbmarliere left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		=cut

		sub find_retry_entry {

Conversation

Avinesh commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification runs:

Uh oh!

Uh oh!

pevik Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

Avinesh Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

pevik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pevik commented Nov 3, 2026

Uh oh!

czerw commented Nov 3, 2026

Uh oh!

czerw left a comment

Choose a reason for hiding this comment

Uh oh!

czerw Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

rbmarliere left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Avinesh commented Nov 2, 2026 •

edited

Loading

pevik left a comment •

edited

Loading