Setup automatic benchmark in CI

[nelimee]

Is your feature request related to a problem? Please describe.

This repository hosts a Python library that is quite performance-hungry and that might scale quite badly with input size. For this reason, we would like the code in this repository to be reasonably optimized.

The problem is that we have no way of tracking the code performance and checking performance regressions/improvements.

Describe the solution you'd like

We need to:

• Introduce a stable bench-marking suite
• Have a way to track historic benchmark data

Describe alternatives you've considered

The bench-marking test suite should likely be composed of several .stim files implementing several QEC codes (e.g., surface code, repetition code, color code, ...) for several distances (from 3 to 11 for example?). As benchmarks might be run often, it would be nice to avoid very large runtime.

Historic benchmark data could be handled with:

• https://github.com/benchmark-action/github-action-benchmark that seems quite nice as it directly uploads benchmarks as a sub-page of an existing GitHub Page website. It also directly integrates with pytest-benchmark.
• https://github.com/bencherdev/bencher that seems a little bit less convenient as it requires a third-party service.
• https://github.com/airspeed-velocity/asv seems like a nicer tool, as it does not have to suffer from Github runner high variance when doing benchmarks.

[epompeii]

@nelimee if you are interested in the relative continuous benchmarking capability provided by asv but you still want to use pytest-benchmark, then you can have your cake and eat it too with Bencher:

• Relative Continuous Benchmarking
• pytest-benchmark adapter

Alternatively, if you want to use asv for the relative continuous benchmarking but still keep track of a historical baseline, Bencher supports that too:

• Track base branch in GitHub Actions
• asv adapter

that seems a little bit less convenient as it requires a third-party service.

Bencher does require a backend to store the data. This has a couple of advantages.

Compared to github-action-benchmark, this prevents you from having to clone all of the historical data on every run. Long term, this can really add up for some users.

Compared to asv, this allows you to use statistics instead of several iterations every run, making your benchmarks run 4-10x faster, depending on setup.

I'm the maintainer of Bencher, so more than happy to help answer any questions.

[nelimee]

@nelimee if you are interested in the relative continuous benchmarking capability provided by asv but you still want to use pytest-benchmark, then you can have your cake and eat it too with Bencher:

* [Relative Continuous Benchmarking](https://bencher.dev/docs/how-to/track-benchmarks/#relative-continuous-benchmarking)

* [`pytest-benchmark` adapter](https://bencher.dev/docs/explanation/adapters/#-python-pytest)

I do not really care about keeping pytest for benchmark. I am fine learning a "new" framework for that (even though in this case this is mostly 2 to 3 rules to follow to let asv know that your code is a benchmark).

Alternatively, if you want to use asv for the relative continuous benchmarking but still keep track of a historical baseline, Bencher supports that too:

* [Track base branch in GitHub Actions](https://bencher.dev/docs/how-to/github-actions/#base-branch)

* [`asv` adapter](https://bencher.dev/docs/explanation/adapters/#-python-asv)

I still need to have asv push on GitHub pages and check how everything is handled, but I do not see yet the benefits of using bencher for us.

that seems a little bit less convenient as it requires a third-party service.

Bencher does require a backend to store the data. This has a couple of advantages.

Compared to github-action-benchmark, this prevents you from having to clone all of the historical data on every run. Long term, this can really add up for some users.

For the moment, this project is small enough. If that really becomes an issue, we might think about it. But if that is not a major issue for numpy, I do not see a world where it is an issue for that repository.

Compared to asv, this allows you to use statistics instead of several iterations every run, making your benchmarks run 4-10x faster, depending on setup.

I do not understand that sentence. Statistics are necessarily computed on several values, obtained through several iterations, what am I missing?

I'm the maintainer of Bencher, so more than happy to help answer any questions.

I really like your doc and that seems to be a nice project! The major blocking point for me is basically that I do not see a real added value for this project to add bencher on top of asv.

[epompeii]

Compared to asv, this allows you to use statistics instead of several iterations every run, making your benchmarks run 4-10x faster, depending on setup.

I do not understand that sentence. Statistics are necessarily computed on several values, obtained through several iterations, what am I missing?

With asv/relative continuous benchmarking you are computing those several iterations every run. Iterations for the base branch + iterations for the candidate branch.

This only allows asv to do a simple percentage check (10% by default) to detect regressions.

I was trying to convey that using statistical continuous benchmarking would allow you to only have to have iterations for the candidate branch, when evaluating a PR.

But if that is not a major issue for numpy, I do not see a world where it is an issue for that repository.

I think they stopped pushing results back in 2022:

• https://pv.github.io/numpy-bench
• CI: Make benchmark asv run quick to only check that benchmarks work numpy/numpy#22694

I'm not sure what the reason was though.

I really like your doc and that seems to be a nice project! The major blocking point for me is basically that I do not see a real added value for this project to add bencher on top of asv.

Thank you for the kind words. I think that makes sense, given your preferences.