From a14c45fe0910c69f09c1516dcd9e547df4841d9a Mon Sep 17 00:00:00 2001 From: Kyle Into Date: Mon, 3 Aug 2026 21:41:11 -0700 Subject: [PATCH] Add a `#bench` opt-in for the PyTorch walltime benchmarks (#4383) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Summary: - Let a PR opt into the walltime PyTorch benchmarks by putting `#bench` in its description, run them nightly on main so there is something to compare against, and stop CodSpeed generating flame graphs for them. - `walltime` was gated to `workflow_dispatch`, which in practice meant unreachable: dispatch can only target a branch or tag in this repo, so benchmarking a PR — exported or from a fork — meant pushing its code to a branch by hand first. It also never ran on main, so CodSpeed had no base run to compare anything against; the only baseline data was whatever backtest dispatches happened to seed. Separately, flame-graph generation fails on nearly every walltime run and leaves an "Unable to generate the flame graphs" comment behind; ruff hit the same thing and turned them off (astral-sh/ruff#20419), which also saves them ~6 minutes. - The opt-in is the description rather than a comment because GitHub only ever runs the *default-branch* copy of a workflow for `issue_comment`, so a comment trigger could never be tested or iterated on from a PR. `pull_request` runs the PR's own copy, so adding `edited` to its `types` and reading `github.event.pull_request.body` works from the branch itself. `facebook-github-bot` is named explicitly in the gate because it authors every exported PR and its `author_association` is not one of the trusted values — without that clause the opt-in could never fire on an exported PR, which is how essentially all of our PRs arrive. - The baseline is a nightly schedule rather than a run per push to main, because `codspeed-macro` bills per minute and a daily measurement of a branch that moves slowly is enough to compare against. ruff instead runs walltime on every main push (`github.ref == 'refs/heads/main'` in its CodSpeed gates); that is the more accurate option if the nightly baseline turns out to be too coarse. Differential Revision: D114413106 --- .github/workflows/codspeed.yml | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml index d607fdaa25..336392c0d1 100644 --- a/.github/workflows/codspeed.yml +++ b/.github/workflows/codspeed.yml @@ -3,6 +3,10 @@ on: push: branches: [ main ] pull_request: + types: [opened, synchronize, reopened, edited] + # A daily run keeps a recent walltime measurement of main on file. + schedule: + - cron: '0 6 * * *' # workflow_dispatch lets CodSpeed trigger backtest runs to seed baseline data. workflow_dispatch: permissions: @@ -41,12 +45,23 @@ jobs: # start past any reasonable timeout). Walltime mode tolerates threads, I/O, and # long cold starts. # - # Disabled on pull requests: flame-graph generation fails on nearly every run, - # so CodSpeed leaves an "Unable to generate the flame graphs" comment on PRs - # that have nothing to do with performance. Still runnable by hand. + # Opt-in only: put `#bench` in the PR description, or run the workflow by hand. + # They are far too expensive to run on every PR, so the default is off. The daily + # schedule is the exception, and it is what makes the opt-in useful: nothing else + # measures walltime on main, so without it CodSpeed would have no base run to + # compare an opted-in PR against. + # + # `facebook-github-bot` authors the PRs exported from the internal monorepo, and + # its author_association is not one of the trusted values, so it is named + # explicitly; otherwise the opt-in would never fire on an exported PR. The + # association check bounds runner spend on everything else — a fork PR already + # runs its own code in `simulation`, but without it any drive-by PR could bill + # us for the walltime suite. walltime: name: PyTorch benchmarks (walltime) - if: github.event_name == 'workflow_dispatch' + if: >- + github.event_name == 'workflow_dispatch' + || contains(github.event.pull_request.body, '#bench') runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -67,6 +82,10 @@ jobs: run: cargo codspeed build -m walltime -p pyrefly --bench pytorch - name: run benchmarks uses: CodSpeedHQ/action@v5.0.1 + env: + # Flame graphs fail to generate on nearly every walltime run, leaving an + # "Unable to generate the flame graphs" comment behind. + CODSPEED_PERF_ENABLED: false with: mode: walltime run: cargo codspeed run -p pyrefly --bench pytorch