Skip to content

ci(forward-merge): notify Slack when the cascade is blocked or fails #3123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maaaathis merged 1 commit into from
Sep 8, 2026
Merged

ci(forward-merge): notify Slack when the cascade is blocked or fails #3123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 45 additions & 2 deletions .github/workflows/forward-merge-drift.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ name: Forward-merge drift check
# survived even the catch-up, and makes it loud. It deliberately neither merges
# nor dispatches anything; recovery is the manual `forward-merge.yml` dispatch,
# already a first-class path. Detection is what was missing.
#
# The escalation is an issue plus a Slack message. Slack mirrors the issue and so
# inherits its idempotence — the schedule runs hourly, the alert must not.

on:
schedule:
Expand All @@ -49,6 +52,7 @@ env:
GH_TOKEN: ${{ secrets.PUBLISH_PAT }}
REPO: ${{ github.repository }}
MAX_DRIFT_MINUTES: ${{ inputs.max_drift_minutes || '120' }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

jobs:
drift:
Expand All @@ -61,6 +65,7 @@ jobs:
fetch-depth: 0

- name: Compare `main` against `next` 🔍
id: drift
run: |
set -euo pipefail

Expand Down Expand Up @@ -163,5 +168,43 @@ jobs:
'' \
'/cc @mfal @Lisa18289 @ins0 @maaaathis @Jan-Eimertenbrink')"

gh issue create --repo "$REPO" --label automated \
--title "$title" --body "$body"
url="$(gh issue create --repo "$REPO" --label automated \
--title "$title" --body "$body")"

# Slack mirrors the ISSUE, so it inherits the issue's idempotence: every
# early exit above — no drift, drift below the threshold, a young
# blocker, an alert already open — leaves this output empty. Only a
# newly opened drift issue reaches the channel, so the hourly schedule
# does not repeat itself.
{
echo "alert<<DRIFT_ALERT_EOF"
printf '%s\n%s\n\n%s\n\n<%s|Drift issue> · <%s|Workflow run>\n' \
'*`next` has fallen behind `main` (forward-merge drift)*' \
"\`next\` is behind \`main\` by ${behind} commit(s), and the oldest un-merged commit is ${age_min} minutes old — past the ${MAX_DRIFT_MINUTES} minute threshold." \
"${cause}" "$url" "$RUN_URL"
echo "DRIFT_ALERT_EOF"
} >> "$GITHUB_OUTPUT"

# `continue-on-error`: the drift issue is the source of truth, Slack is the
# additional signal. A webhook outage must not fail the check that just
# reported real drift.
- name: Notify Slack about the drift 🚨
if: steps.drift.outputs.alert != ''
continue-on-error: true
uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
webhook-type: incoming-webhook
payload: |
{
"text": "next has fallen behind main (forward-merge drift)",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ${{ toJSON(steps.drift.outputs.alert) }}
}
}
]
}
83 changes: 75 additions & 8 deletions .github/workflows/forward-merge.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@ name: Forward-merge main into next
# code conflict escalates to a sync ISSUE (ADR 0004 §4) — the resolution itself
# happens in a developer checkout via `pnpm sync:resolve`, because that is the
# only place the merge drivers run. Any other failure opens an issue too
# (ADR 0004 §10).
# (ADR 0004 §10). Both escalations additionally post to Slack — see the
# `record_escalation` comment below for why the conflict path cannot key its
# notification on `failure()`.
#
# The push to `next` triggers `publish.yml`, which re-derives the version and
# publishes the `next` prerelease (ADR 0004 §6). This workflow itself stays
Expand Down Expand Up @@ -67,6 +69,7 @@ concurrency:
env:
GH_TOKEN: ${{ secrets.PUBLISH_PAT }}
REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

jobs:
forward-merge:
Expand Down Expand Up @@ -142,6 +145,22 @@ jobs:
# "Close the escalation" step below acts on it. See the comment there
# for why that had to move out of this step.

# Slack mirror of the escalation below (ADR 0004 §4). It keys on the
# ESCALATION, not on the run: the conflict path exits 0, so a
# `failure()` notification would never see a blocked cascade. And it
# fires only where `open_sync_issue` itself speaks up — each of its
# quiet `return 0`s, taken when the blockage is already reported,
# leaves this output empty, so a cascade that stays blocked does not
# re-ping the channel on every push to `main`.
record_escalation() {
{
echo "escalation<<ESCALATION_EOF"
printf '%s\n%s\n\n<%s|Escalation> · <%s|Workflow run>\n' \
'*Forward-merge blocked: `main` into `next`*' "$1" "$2" "$RUN_URL"
echo "ESCALATION_EOF"
} >> "$GITHUB_OUTPUT"
}

# Escalation (ADR 0004 §4). The cascade does NOT create a branch or
# a pull request here: a PR is a change proposal, and until a human has
# resolved the conflict there is nothing to propose. It opens an issue
Expand All @@ -155,7 +174,7 @@ jobs:
# git, and the pull request that eventually appears is a clean,
# reviewable merge.
open_sync_issue() {
local conflicted body title existing open_pr fingerprint marker refresh
local conflicted body title existing open_pr fingerprint marker refresh url
title="Forward-merge blocked: main into next needs a manual resolution"

conflicted="$(git ls-files --unmerged | cut -f2 | sort -u | sed 's/^/- `/; s/$/`/')"
Expand Down Expand Up @@ -196,8 +215,9 @@ jobs:
echo "::notice::Sync PR #${open_pr} already reports this exact blockage — staying quiet."
return 0
fi
gh pr comment "$open_pr" --repo "$REPO" --body "$refresh"
url="$(gh pr comment "$open_pr" --repo "$REPO" --body "$refresh")"
echo "::warning::Sync PR #${open_pr} does not cover the current blockage — commented."
record_escalation "Sync PR #${open_pr} does not cover the current blockage — commented. \`main\` is at \`${SHORT_SHA}\`." "$url"
return 0
fi

Expand All @@ -209,8 +229,9 @@ jobs:
echo "::notice::Sync issue #${existing} already reports this exact blockage — not commenting again."
return 0
fi
gh issue comment "$existing" --repo "$REPO" --body "$refresh"
url="$(gh issue comment "$existing" --repo "$REPO" --body "$refresh")"
echo "::warning::Sync issue #${existing} refreshed — the blockage changed."
record_escalation "Sync issue #${existing} was refreshed — the cascade is blocked on a different conflict than the one reported there." "$url"
return 0
fi

Expand Down Expand Up @@ -240,8 +261,9 @@ jobs:
'' \
"${marker}")"

gh issue create --repo "$REPO" --label sync --label automated \
--title "$title" --body "$body"
url="$(gh issue create --repo "$REPO" --label sync --label automated \
--title "$title" --body "$body")"
record_escalation 'The merge hit a genuine code conflict. The cascade is paused: until someone runs `pnpm sync:resolve`, every later commit on `main` stays out of `next`.' "$url"
}

git fetch --no-tags origin main
Expand Down Expand Up @@ -304,6 +326,32 @@ jobs:
echo "::error::Retry re-merge/push failed."
exit 1

# `continue-on-error` keeps Slack out of the cascade's verdict. A webhook
# outage or a missing secret must not turn the conflict path — an expected
# non-error path — into a run failure, which the §10 step below would then
# report as a genuine forward-merge failure and open an issue for. The
# sync issue stays the source of truth; Slack is the additional signal.
- name: Notify Slack that the cascade is blocked 💬
if: steps.merge.outputs.escalation != ''
continue-on-error: true
uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
webhook-type: incoming-webhook
payload: |
{
"text": "Forward-merge blocked: main into next",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ${{ toJSON(steps.merge.outputs.escalation) }}
}
}
]
}

# The ONE place that clears an escalation (ADR 0004 §4). It fires whenever
# the cascade came out healthy — `next` carries `main`'s CODE — which is
# true in three cases:
Expand Down Expand Up @@ -349,8 +397,6 @@ jobs:
# 0004 §10).
- name: Open an issue on failure 🚨
if: failure() && steps.guard.outputs.pending == 'true'
env:
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
gh label create automated --repo "$REPO" --color EDEDED \
Expand All @@ -365,3 +411,24 @@ jobs:
--title "Forward-merge main into next failed" \
--body "$body" \
|| echo "::warning::Could not open the failure issue."

- name: Notify Slack about the failure 🚨
if: failure() && steps.guard.outputs.pending == 'true'
continue-on-error: true
uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
webhook-type: incoming-webhook
payload: |
{
"text": "Forward-merge main into next failed",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Forward-merge `main` into `next` failed.*\nNot a merge conflict (ADR 0004 §10) — `next` may now be falling behind `main`. An issue was opened.\n\n<${{ env.RUN_URL }}|Workflow run>"
}
}
]
}
16 changes: 13 additions & 3 deletions docs/adr/0004-forward-merge-main-into-next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,15 @@ The escalation is **idempotent**: while a sync issue or a sync PR is open, the
cascade does not open a second one, and it never touches the branch a human may
be working on.

The escalation is **additionally posted to Slack** (`SLACK_WEBHOOK_URL`). It has
to key on the escalation, not on the run: a conflict is the expected non-error
path, the run exits 0, and a `failure()` notification would never see a blocked
cascade. Keying on the escalation also inherits its idempotence — a cascade that
stays blocked does not re-ping the channel on every push to `main`. The Slack
step is `continue-on-error`: the issue is the source of truth, and a webhook
outage must not turn the conflict path into a run failure that §10 would then
report as a genuine forward-merge failure.

**The sync PR must be merged as a true merge commit** (`main` stays an ancestor
of `next`). Squash/rebase-merging it would reintroduce trap #1. This is enforced
by branch protection on `next` (see §5), consistent with §1.
Expand Down Expand Up @@ -261,7 +270,8 @@ Two mitigations, both implemented:
schedule and escalates when the gap outlives a threshold, ignoring drift that
an open sync issue or sync PR already accounts for — and ageing those too, so
a forgotten escalation cannot block the cascade indefinitely without saying
so.
so. Its escalation goes to Slack as well, mirroring the issue and therefore
its idempotence: the schedule runs hourly, the alert does not.

Giving `forward-merge.yml` its own group was considered and rejected:
forward-merge runs evicting each other is harmless (they always merge `main`'s
Expand Down Expand Up @@ -349,8 +359,8 @@ bypass actor on `next`. No new GitHub App is introduced for now.
A **conflict** produces a sync PR (§4) — that is the expected, non-error path. A
**non-conflict failure** (fetch-before-push retry exhausted, an error in the
workflow itself, a failing generator) is different: the workflow **opens an
issue and pings CODEOWNERS**, so a silently-failing sync cannot let `next`
quietly fall behind `main`.
issue and pings CODEOWNERS**, and posts the same alert to Slack, so a
silently-failing sync cannot let `next` quietly fall behind `main`.

### 11. The major line

Expand Down
Loading