ceph-perf: tighten PR regression gate#2566
Conversation
25b9570 to
18e33c0
Compare
There was a problem hiding this comment.
Pull request overview
Updates the ceph-perf pull request performance job to behave as a stricter release-style regression gate by pinning the benchmark definition to ceph-main and surfacing regression status directly on GitHub PRs.
Changes:
- Parameterize CBT workload selection in the
run-cbtmacro and pin the benchmark YAML toceph-main. - Add/refresh GitHub PR status context strings to make perf regression results visible in the PR UI.
- Re-enable the perf PR job template (
disabled: false) and clarify the job’s purpose via description.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
18e33c0 to
c517923
Compare
Matan-B
left a comment
There was a problem hiding this comment.
Few pieces still missing here. I’m not sure this would work with older compilers and exiting CBT support. I would suggest trying to run some of the tests locally first to see if the numbers make sense before we start blocking PRs with this job.
Is this check required once enabled?
We should probably also add some details to Crimson’s dev guide, how the test works, when a PR is blocked (under which cases) and where the results are stored. This should help devs mitigate failures.
c517923 to
7359ce6
Compare
There was a problem hiding this comment.
I'm not sure I understand if this is ready for review or was tested locally.
If it was, can you please share examples of how the results comparison work?
The clang version that was already mentioned in the previous review would still FTBFS.
CBT is still not updated for the relevant usage with Crimson.
Under which cases a PR is considered blocked (dev docs)?
It is also not clear what the strategy is here - is this intended to run only against Crimson PRs?
Please make sure there is a solid plan for how this is going to be implemented. Starting with a draft PR is also a good approach, but please be clear about the draft stage (what is missing, what is not yet tested/verified and what the next steps are or what is working so far). Otherwise, this PR does not seem ready for review.
My suggestion here would be to enable this check as a non-required PR check and iterate on it until it can be re-enabled as a required one. Alternatively, running parts of the jobs locally to make sure all is working as expected.
I would be happy to help in any way if you have any questions.
94866b0 to
064d528
Compare
064d528 to
007d18b
Compare
f2cd7b4 to
76ddf37
Compare
This describes the perf PR job as a release regression check, pin the benchmark definition to ceph-main, and surface the perf regression status in GitHub. This also reenables the ceph-perf-pull-request Fixes: https://tracker.ceph.com/issues/75665 Signed-off-by: Kautilya Tripathi <kautilya.tripathi@ibm.com>
76ddf37 to
9103af4
Compare
This describes the perf PR job as a release regression check, pin the benchmark definition to ceph-main, and surface the perf regression status in GitHub. This also reenables the ceph-perf-pull-request
Fixes: https://tracker.ceph.com/issues/75665