Skip to content

ci: run every suite on Dependabot PRs and deploy their review apps - #3079

Open
mfal wants to merge 3 commits into
mainfrom
claude/dependabot-mrs-review-apps-tests-17e0bd
Open

ci: run every suite on Dependabot PRs and deploy their review apps#3079
mfal wants to merge 3 commits into
mainfrom
claude/dependabot-mrs-review-apps-tests-17e0bd

Conversation

@mfal

@mfal mfal commented Sep 2, 2026

Copy link
Copy Markdown
Member

Two gaps on a Dependabot PR, both rooted in the same fact: secrets.* in a Dependabot-triggered run resolves against the repository's Dependabot secret store, not the Actions one.

Review apps

The images were always built and pushed — only the deploy job was skipped. cleanup-previews.yml had it worse: it ran without a guard and failed on every Dependabot merge with Missing required environment variables, after the merge, where nobody looks.

  • deploy-previews-dependabot.yml (new) — workflow_run on Build & Deploy Preview Apps, runs from the default branch where the secrets are ordinary, and posts the preview URLs.
  • cleanup-previews-dependabot.yml (new) — the same pattern behind Cleanup Preview Apps.
  • Neither ever checks out the PR's code. The PR reaches them only as the two images its build job already pushed, which is what makes the write token safe to hold there.
  • The actor conditions are the exact complement of the guards in the two ordinary preview workflows, so a push a person makes onto a dependabot/* branch is still deployed there — not a second time here.
  • cleanup-previews.yml skips its steps, not its job, for that actor: a job-level if concludes the whole run skipped, and only a run that actually completes is a dependable trigger.

Tests

The full cross-version suite — both harnesses against every published version — was the one suite a Dependabot PR never ran (skipping on all four open ones). It was label-only, while visual has run unconditionally for a while. A dependency bump is exactly the change that moves rendered structure without touching a component.

  • test-cross-version-label.yml now also runs on opened/reopened/synchronize for Dependabot PRs.
  • dependabot-auto-merge.yml waits for both suites. Neither is a required status check in the ruleset, so this workflow is the only thing between a red run and a merge.

The gate reads each suite's verdict back for the PR head instead of trusting the payload, ordered by run id with cancelled/skipped dropped. That ordering is not pedantry: one commit carries four to five runs of the same suite, because Dependabot attaches its own labels right after opening the PR and every labeled event starts one that the concurrency group then cancels. Their created_at are equal to the second, and the surviving run is not necessarily the highest id.

Corrected rationale

Several of these workflows claimed a Dependabot-triggered run gets a read-only GITHUB_TOKEN. The run log says PullRequests: write, and the ghcr push works — only the secrets are withheld. Comments fixed in four workflows (no behaviour change there), and CONTRIBUTE.md now records both halves of what such a run does and does not get.

Verifying this

Like every workflow_run workflow, the two new ones are inert until they land on main — they cannot be exercised from this PR. After merge, @dependabot recreate on one of the open bump PRs produces a push → build → deploy chain to watch.

Cost

The cross-version suite is unsharded (~25 min, 45 min timeout) and now runs on every Dependabot PR, plus the cancelled label-storm runs. At four PRs a week with the rebase cascade, that is the most noticeable part of this change.

@mfal mfal self-assigned this Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for ./packages/components/

Status Category Percentage Covered / Total
🔵 Lines 78.69% 746 / 948
🔵 Statements 78.57% 763 / 971
🔵 Functions 80.09% 165 / 206
🔵 Branches 70.33% 377 / 536
File CoverageNo changed files found.
Generated in workflow #6659 for commit b13a0ab by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

Preview environments are ready:

Type URL
docs pr-3079.docs.review.flow-components.de
storybook pr-3079.storybook.review.flow-components.de

Images:

  • docs: ghcr.io/mittwald/flow/docs:pr-3079
  • storybook: ghcr.io/mittwald/flow/storybook:pr-3079

@mfal
mfal marked this pull request as ready for review September 2, 2026 09:24
@mfal
mfal requested a review from a team September 2, 2026 09:24
Two gaps on a Dependabot PR, both rooted in the same fact: `secrets.*` in a
Dependabot-triggered run resolves against the repository's *Dependabot* secret
store, not the Actions one.

**Review apps.** The images were always built and pushed; only the deploy job
was skipped, and `cleanup-previews.yml` failed on every Dependabot merge
("Missing required environment variables", after the merge, where nobody
looks). Both halves now run in a `workflow_run` from the default branch, which
has the secrets — the same mechanism `dependabot-auto-merge.yml` uses. Neither
new workflow ever checks out the PR's code: the PR reaches them only as the two
images its build job pushed. The actor conditions are the exact complement of
the guards in the two ordinary preview workflows, so a push a person makes onto
a `dependabot/*` branch is still deployed there and not a second time here. The
cleanup workflow skips its *steps* rather than its job for that actor: a
job-level `if` concludes the run `skipped`, and only a run that completes is a
dependable trigger.

**Tests.** The full cross-version suite (both harnesses, every published
version) was the one suite a Dependabot PR never ran — it was label-only, while
visual has run unconditionally for a while. A dependency bump is exactly the
change that moves rendered structure without touching a component, so it now
runs there too, and `dependabot-auto-merge.yml` waits for both suites. Neither
is a required status check, so this workflow is the only thing between a red run
and a merge.

The gate reads the verdicts back per workflow instead of trusting the payload,
ordered by run `id` with `cancelled`/`skipped` dropped. One commit carries four
to five runs of the same suite: Dependabot attaches its own labels right after
opening the PR, each `labeled` event starts a run, and the concurrency group
cancels the rest — with `created_at` equal to the second, and the surviving run
not necessarily the highest id.

Also corrects the rationale several of these workflows carried. They claimed a
Dependabot-triggered run gets a read-only `GITHUB_TOKEN`; the run log says
`PullRequests: write`, and the ghcr push works. Only the secrets are withheld.
CONTRIBUTE.md now records both halves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mfal
mfal force-pushed the claude/dependabot-mrs-review-apps-tests-17e0bd branch from 367777e to bd0a141 Compare September 2, 2026 11:53
mfal and others added 2 commits September 3, 2026 09:02
One conflict, in `.github/workflows/cleanup-previews.yml`: the actions group
bump (#3069) re-pinned `actions/checkout` and `pnpm/action-setup` on the very
lines where this branch adds the `CLEAN_UP_HERE` step guards. Kept both intents
— the guards, on the new pins.

The two workflows this branch adds could not conflict, because main has never
seen them, so they still carried the pre-#3069 shas. Carried the new pins over
by hand: leaving them behind would have made this PR reintroduce the exact
versions #3069 just replaced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant