Skip to content

Issue #340: Add benchmark env smoke test to CI#343

Open
Sammy-Dabbas wants to merge 1 commit into
AVSLab:developfrom
Sammy-Dabbas:issue-340-benchmark-ci
Open

Issue #340: Add benchmark env smoke test to CI#343
Sammy-Dabbas wants to merge 1 commit into
AVSLab:developfrom
Sammy-Dabbas:issue-340-benchmark-ci

Conversation

@Sammy-Dabbas

Copy link
Copy Markdown
Contributor

Closes #340.

What this does

Adds a run-benchmarks job to commit_checks.yml plus a small benchmarks/smoke_test.py driver. The job builds via the existing .github/actions/build action (Python 3.11, pip-options: all,rllib, mirroring check-doc-build) and then constructs one representative environment from each benchmark module -- nadir_science, aeos_single, and rso_inspection -- and steps each 3 times.

Design choices

  • Smoke test, not full benchmark. benchmark.py --total_timesteps 0 still spins up Ray and builds a full PPO model, so the smoke script bypasses Ray/RLlib entirely and exercises only environment construction and the PettingZoo step loop. Breakages in benchmark env definitions fail in minutes instead of requiring a training run.
  • env_args carries the RLlib-only episode_data_callback/satellite_data_callback keys that ConstellationTasking.__init__ does not accept; the script strips them the same way _constellation_tasking_env_creator does.
  • The script runs with working-directory: benchmarks because the benchmark modules use top-level imports (from benchmark import BenchmarkEnv).
  • Single Python version (3.11) rather than the full pytest matrix to keep CI cost low.

Verification

Basilisk is a heavy native build, so locally I validated the workflow YAML, ruff/isort cleanliness, and traced every invocation against the actual signatures (ConstellationTasking kwargs, PettingZoo agent-keyed step dicts). The end-to-end run happens in this PR's own CI via the new job. Happy to adjust scope (e.g. more envs per module or a different Python version) if you'd prefer.

Adds a run-benchmarks CI job that builds one representative environment
from each benchmark module (nadir_science, aeos_single, rso_inspection)
and steps it a few times, so PRs that break benchmark env construction
fail fast without running full PPO training.
@Sammy-Dabbas Sammy-Dabbas force-pushed the issue-340-benchmark-ci branch from 9cb65f7 to 9b92df3 Compare July 4, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add benchmark envs to CI pipeline

1 participant