Improve ref sim ci

[burdorfmitchell]

PR for allowing a rerun of just the failed jobs for the compare-post-benchmark workflow. Also bumps regression failure to 150%. Also attempts to fix the mpi errors being seen on ci for running reference simulations. Also swaps to downloading gleam from the Brown Digital Repository.

Description

Motivation and Context

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Reference simulation update or replacement
• Documentation change (documentation changes only)
• Version change
• Build or continuous integration change

Checklist:

For all pull requests:

• I have read the contribution guide.
• My code follows the code style of this project.

New feature checklist:

• I have added or updated the docstrings associated with my feature using the numpy docstring format.
• I have updated the documentation to highlight my new feature (if appropriate).
• I have added tests to cover my new feature.
• All new and existing tests pass.
• I have checked that I reproduce the reference simulations or if there are differences they are explained below (if appropriate). If there are changes that are correct, I will update the reference simulation files after this PR is merged.
• I have checked (e.g., using the benchmarking tools) that this change does not significantly increase typical runtimes. If it does, I have included a justification in the comments on this PR.
• I have updated the CHANGELOG.

Build or continuous integration change checklist:

• If required or optional dependencies have changed (including version numbers), I have updated the readme to reflect this.
• If this is a new CI setup, I have added the associated badge to the readme and to references/make_index.py (if appropriate).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (256cc87) to head (d2feca2).

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #632   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        11           
  Lines         2455      2455           
=========================================
  Hits          2455      2455           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bhazelton]

Looks good to me, thanks!