Skip to content

test: start the test repo once to fix flaky setup_all failures - #225

Merged
zachdaniel merged 1 commit into
ash-project:mainfrom
Forge-of-Apps:fix-flaky-repo-setup-all
Oct 20, 2026
Merged

zachdaniel merged 1 commit into
ash-project:mainfrom
Forge-of-Apps:fix-flaky-repo-setup-all

Conversation

@etareduction

Copy link
Copy Markdown
Contributor

CI on main intermittently ends with N tests, 0 failures, 2–5 invalid: one module's setup_all fails with

** (RuntimeError) could not lookup Ecto repo AshOban.Test.Repo because it was not started or it does not exist
  ...
  (oban 2.24.1) lib/oban/migration.ex:212: Oban.Migration.verify_migrated!/1
  (oban 2.24.1) lib/oban.ex:599: Oban.start_link/1

(e.g. runs 39450958371, 38788849819, 38503023987, 33346691045, 33293567334). The module varies.

Seven test modules call AshOban.Test.Repo.start_link() in setup_all and ignore the result. The repo gets linked to the setup_all process of whichever module started it first, and ExUnit only stops that process's linked children asynchronously after the module finishes. The next module's start_link() can return {:error, {:already_started, _}} just before the old repo goes down, and then Oban.start_link/1 finds no repo.

This starts the repo once in test_helper.exs and drops the per-module calls.

Reproducing

--repeat-until-failure runs the suite repeatedly in one VM. VerifyUseTenantFromRecordTest defines modules inside its tests and fails on any second iteration, so it has to be excluded:

mix test --seed 564897 --repeat-until-failure 500 --exclude module:AshOban.Verifiers.VerifyUseTenantFromRecordTest

On main, 11 out of 11 trials (seeds 564897 and 1–10) produced invalid modules, after 12–289 iterations (median 31). Across 992 iterations that is about 1 in 90, so 500 repeats miss it about 0.4% of the time. With this change, seeds 564897 and 1–3 ran all 501 iterations each (2004 in total) with no failures or invalid modules.

@zachdaniel
zachdaniel merged commit 0ecba99 into ash-project:main Oct 20, 2026
23 of 25 checks passed
@zachdaniel

Copy link
Copy Markdown
Contributor

🚀 Thank you for your contribution! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants