Skip to content

Commit 8e58395

Browse files
bradygasterCopilot
andcommitted
Document v0.8.22 release incident failures
Added Failure 3 to Kobayashi's charter documenting: - Invalid 4-part semver (0.8.21.4) committed and tagged - GitHub Release created as draft (no workflow trigger) - NPM_TOKEN type not verified (user token blocked publish) Updated history.md with detailed incident report: - Root causes for each failure - Lessons learned (semver rules, draft vs published, token types) - Pre-flight checklist for future releases - State corruption assessment and recovery steps Created decision proposal for release guardrails: - Pre-publish semver validation in publish.yml - GitHub Release draft prevention/verification - NPM_TOKEN type validation - Release runbook skill with checklists and rollback procedures This was my failure. These guardrails prevent recurrence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 05573c7 commit 8e58395

3 files changed

Lines changed: 337 additions & 1 deletion

File tree

‎.squad/agents/kobayashi/charter.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,21 @@ Learn from these past failures:
107107
- **Root cause:** When the easy path failed, I took the easiest exit (close) instead of investigating alternatives (fetch from fork, manual merge, conflict resolution strategies).
108108
- **Prevention:** NEVER close a PR when asked to merge. Merge failures require investigation and resolution, not abandonment.
109109

110-
**Pattern:** When git operations get complicated, I historically took the easiest path instead of the correct path. This ends now.
110+
**Failure 3 — Invalid semver and draft release (v0.8.22 incident):**
111+
- **What happened:** Brady asked for v0.8.22 release. I committed version 0.8.21.4 to main, created a tag v0.8.21.4, and created a GitHub Release as a DRAFT.
112+
- **What went wrong:**
113+
1. 4-part version numbers (0.8.21.4) are INVALID semver for npm — npm mangled it to 0.8.2-1.4
114+
2. Created GitHub Release as DRAFT instead of published — draft releases do NOT trigger `release: published` webhook events, so the publish workflow never ran
115+
3. Did not verify NPM_TOKEN was an automation token before publishing — it was a user token with 2FA enabled, blocking automated publish
116+
4. Required multiple corrections from Brady before getting the release right
117+
- **Root cause:** Failed to validate semver format before committing. Failed to understand GitHub Release draft vs. published semantics. Failed to verify token type before publish attempt. No pre-flight validation checklist.
118+
- **Prevention:**
119+
1. **ALWAYS validate semver:** 3-part versions only (X.Y.Z or X.Y.Z-prerelease). Never 4-part versions. Validate with `npm version` or semver library before committing.
120+
2. **NEVER create draft releases:** Use `gh release create --draft=false` or explicitly publish after creation. Draft releases are invisible to automation.
121+
3. **Verify NPM_TOKEN type:** Before first npm publish, verify token is "Automation" type (not "Publish" with 2FA). Check token capabilities at https://www.npmjs.com/settings/{user}/tokens
122+
4. **Pre-publish checklist:** (1) semver valid? (2) token type correct? (3) release published (not draft)? (4) workflow triggered?
123+
124+
**Pattern:** When releasing under pressure, I skip validation steps and create invalid state. All releases now require pre-flight validation checklist.
111125

112126
## Boundaries
113127

‎.squad/agents/kobayashi/history.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -316,3 +316,107 @@ Brady requested automated npm publishing instead of manual local publishes. Asse
316316
- Rollback strategy: Manual unpublish via `npm unpublish` if issues detected within 72 hours
317317

318318
**Key Learning:** Automated npm publishing reduces human error (version mismatches, forgotten packages, incorrect tag) and provides audit trail via GitHub Actions logs. Provenance attestation strengthens supply chain security.
319+
320+
---
321+
322+
## 📌 INCIDENT REPORT: v0.8.22 Release Failures (2026-03-XX)
323+
324+
**STATUS:** Multiple critical failures. Brady furious. Documented for prevention.
325+
326+
### Failures Committed
327+
328+
**1. Invalid semver — 0.8.21.4 committed to main:**
329+
- Used 4-part version number (0.8.21.4) instead of 3-part semver (0.8.22)
330+
- npm does NOT support 4-part versions — it mangled to 0.8.2-1.4
331+
- Created git tag v0.8.21.4 for invalid version
332+
- Committed to main branch without validation
333+
- **ROOT CAUSE:** No semver validation before version changes. Assumed any version format was valid.
334+
335+
**2. Draft release created instead of published:**
336+
- Created GitHub Release in DRAFT state
337+
- Draft releases do NOT trigger `release: published` webhook events
338+
- Publish workflow (`publish.yml`) never fired because trigger condition not met
339+
- **ROOT CAUSE:** Did not understand GitHub Release draft vs. published semantics. Draft releases are invisible to automation.
340+
341+
**3. NPM_TOKEN type not validated:**
342+
- Did not verify NPM_TOKEN secret was an automation token before publish attempt
343+
- Token was a user token with 2FA enabled
344+
- All publish attempts failed with EOTP error (one-time password required)
345+
- **ROOT CAUSE:** No pre-publish token verification step. Assumed any NPM_TOKEN would work.
346+
347+
**4. Required multiple corrections from Brady:**
348+
- Brady had to intervene multiple times to fix invalid state
349+
- Each correction revealed another unvalidated assumption
350+
- Release process took hours instead of minutes
351+
- **ROOT CAUSE:** No pre-flight checklist. Released under pressure without validation.
352+
353+
### Lessons Learned — Hard Rules
354+
355+
**1. Semver is ALWAYS 3-part for npm:**
356+
- Valid: `X.Y.Z` (e.g., 0.8.22)
357+
- Valid: `X.Y.Z-prerelease.N` (e.g., 0.8.22-preview.1)
358+
- **INVALID:** `X.Y.Z.N` (4-part versions) — npm does not support this
359+
- **Validation:** Use `npm version {version} --no-git-tag-version` to test before committing
360+
- **Prevention:** Add semver validation to `publish.yml` pre-publish step
361+
362+
**2. GitHub Release draft vs. published:**
363+
- **DRAFT:** Release exists but is NOT visible. Does NOT trigger `release: published` event.
364+
- **PUBLISHED:** Release is visible and triggers `release: published` webhook.
365+
- **Automation requirement:** Workflows using `on: release: published` ONLY fire when release is published, NOT when draft is created.
366+
- **Prevention:** Use `gh release create --draft=false` or explicitly publish with `gh release edit {tag} --draft=false`
367+
368+
**3. NPM_TOKEN must be automation token:**
369+
- **User tokens with 2FA:** Cannot be used in CI/CD — require interactive OTP
370+
- **Automation tokens:** Bypass 2FA for CI/CD use (legacy, still supported)
371+
- **Granular access tokens:** Modern alternative, no 2FA, scoped permissions
372+
- **Verification:** Check token type at https://www.npmjs.com/settings/{user}/tokens before configuring CI
373+
- **Prevention:** Document token type requirements in release runbook
374+
375+
**4. Pre-flight checklist required for all releases:**
376+
- [ ] Version is valid semver (3-part only)?
377+
- [ ] Version matches across all package.json files (root, squad-cli, squad-sdk)?
378+
- [ ] NPM_TOKEN is automation token (not user with 2FA)?
379+
- [ ] GitHub Release will be published (not draft)?
380+
- [ ] Workflow trigger conditions will be met?
381+
- **Prevention:** Add pre-publish validation step to `publish.yml`
382+
383+
### State Corruption
384+
385+
**git state:**
386+
- Invalid tag v0.8.21.4 created on main branch
387+
- Invalid version 0.8.21.4 committed to package.json files
388+
- **Recovery:** Tag and commit must be removed/reverted; correct version committed
389+
390+
**npm state:**
391+
- No packages published under invalid version (npm rejected 0.8.2-1.4)
392+
- **Recovery:** No npm cleanup needed
393+
394+
**GitHub state:**
395+
- Draft release may still exist for v0.8.21.4
396+
- **Recovery:** Delete draft release if present
397+
398+
### Prevention — New Guardrails
399+
400+
**Proposed in decision:** `.squad/decisions/inbox/kobayashi-release-guardrails.md`
401+
402+
1. **Pre-publish semver validation:**
403+
- Add validation step in `publish.yml` that verifies version format before npm publish
404+
- Script validates: (1) 3-part format, (2) matches semver spec, (3) not already published
405+
406+
2. **GitHub Release creation verification:**
407+
- Enforce `--draft=false` flag in release creation
408+
- Add workflow check that verifies release is published before proceeding to npm publish
409+
410+
3. **NPM_TOKEN type verification:**
411+
- Document token requirements in README
412+
- Add workflow step that tests token validity before publish (dry-run)
413+
- Add token type check to pre-publish validation
414+
415+
4. **Release runbook:**
416+
- Document complete release process with pre-flight checklist
417+
- Include rollback procedures for each failure mode
418+
- Store in `.squad/skills/release-process/SKILL.md`
419+
420+
### Accountability
421+
422+
**This was my failure.** I rushed the release, skipped validation, and created invalid state. Brady had to fix my mistakes. These guardrails make this failure mode impossible to repeat.
Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
# Release Guardrails — v0.8.22 Incident Prevention
2+
3+
**Date:** 2026-03-XX
4+
**Proposed by:** Kobayashi (Git & Release)
5+
**Context:** v0.8.22 release incident — multiple failures due to missing validation
6+
7+
## Problem
8+
9+
The v0.8.22 release attempt exposed critical gaps in the release validation process:
10+
11+
1. **Invalid semver committed:** 4-part version (0.8.21.4) committed to main — npm mangled it to 0.8.2-1.4
12+
2. **Draft release created:** GitHub Release created as draft — did not trigger `release: published` event, workflow never ran
13+
3. **NPM_TOKEN type not verified:** User token with 2FA blocked automated publish (EOTP error)
14+
4. **Multiple corrections required:** Brady had to intervene repeatedly to fix invalid state
15+
16+
**Root cause:** No pre-flight validation checklist. Released under pressure without verifying preconditions.
17+
18+
## Proposed Guardrails
19+
20+
### 1. Pre-Publish Semver Validation
21+
22+
**Add validation step to `publish.yml` workflow:**
23+
24+
```yaml
25+
- name: Validate semver format
26+
run: |
27+
VERSION="${{ github.event.release.tag_name || inputs.version }}"
28+
VERSION="${VERSION#v}" # Strip 'v' prefix if present
29+
30+
# Validate 3-part semver format (X.Y.Z or X.Y.Z-prerelease)
31+
if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?$'; then
32+
echo "❌ Invalid semver format: $VERSION"
33+
echo "✅ Valid formats: X.Y.Z or X.Y.Z-prerelease.N"
34+
echo "❌ Invalid formats: X.Y.Z.N (4-part versions not supported by npm)"
35+
exit 1
36+
fi
37+
38+
# Validate version matches package.json
39+
PKG_VERSION=$(node -p "require('./package.json').version")
40+
if [ "$VERSION" != "$PKG_VERSION" ]; then
41+
echo "❌ Version mismatch: tag=$VERSION, package.json=$PKG_VERSION"
42+
exit 1
43+
fi
44+
45+
echo "✅ Version $VERSION is valid semver"
46+
```
47+
48+
**Benefits:**
49+
- Catches 4-part versions before npm publish
50+
- Validates version matches package.json
51+
- Fails fast with clear error message
52+
53+
### 2. GitHub Release Draft Prevention
54+
55+
**Option A — Enforce `--draft=false` in creation:**
56+
```bash
57+
gh release create "v${VERSION}" \
58+
--title "v${VERSION}" \
59+
--notes-file CHANGELOG.md \
60+
--draft=false # Explicit non-draft flag
61+
```
62+
63+
**Option B — Add verification step after creation:**
64+
```yaml
65+
- name: Verify release is published
66+
run: |
67+
TAG="${{ github.event.release.tag_name }}"
68+
DRAFT=$(gh release view "$TAG" --json isDraft --jq '.isDraft')
69+
if [ "$DRAFT" = "true" ]; then
70+
echo "❌ Release $TAG is still a draft"
71+
echo "Publishing release..."
72+
gh release edit "$TAG" --draft=false
73+
fi
74+
```
75+
76+
**Benefits:**
77+
- Ensures `release: published` event fires
78+
- Catches accidental draft creation
79+
- Self-correcting (Option B)
80+
81+
**Recommendation:** Use Option A (explicit flag) + Option B (verification) for defense in depth.
82+
83+
### 3. NPM_TOKEN Type Verification
84+
85+
**Add token validation step to `publish.yml`:**
86+
87+
```yaml
88+
- name: Validate NPM token type
89+
run: |
90+
# Test token with dry-run publish
91+
npm publish --dry-run --access public 2>&1 | tee npm-test.log
92+
93+
# Check for 2FA/OTP error
94+
if grep -q "EOTP" npm-test.log || grep -q "one-time password" npm-test.log; then
95+
echo "❌ NPM_TOKEN requires 2FA/OTP — cannot be used in CI/CD"
96+
echo "✅ Required: Automation token or Granular access token"
97+
echo "📝 Create token at: https://www.npmjs.com/settings/bradygaster/tokens"
98+
exit 1
99+
fi
100+
101+
echo "✅ NPM_TOKEN is valid for automated publishing"
102+
```
103+
104+
**Benefits:**
105+
- Detects user tokens with 2FA before publish attempt
106+
- Fails with actionable error message
107+
- Zero risk (dry-run only)
108+
109+
**Alternative:** Document token requirements in README and trust setup. (Less safe but simpler.)
110+
111+
### 4. Release Runbook Skill
112+
113+
**Create `.squad/skills/release-process/SKILL.md`:**
114+
115+
```markdown
116+
# Release Process Skill
117+
118+
## Pre-Flight Checklist
119+
120+
Before starting a release:
121+
122+
- [ ] Version is valid 3-part semver (X.Y.Z or X.Y.Z-prerelease.N)?
123+
- [ ] Version matches across all package.json files?
124+
- [ ] NPM_TOKEN secret is automation token (not user with 2FA)?
125+
- [ ] Will create GitHub Release as PUBLISHED (not draft)?
126+
- [ ] All tests passing on main/dev branch?
127+
- [ ] CHANGELOG.md updated for this version?
128+
129+
## Release Steps
130+
131+
1. **Version bump:** Commit new version to package.json files
132+
2. **Tag creation:** `git tag -a v{VERSION} -m "Release v{VERSION}"`
133+
3. **Push tag:** `git push origin v{VERSION}`
134+
4. **GitHub Release:** `gh release create v{VERSION} --draft=false --notes-file CHANGELOG.md`
135+
5. **Wait for publish:** Monitor workflow at https://github.com/bradygaster/squad/actions
136+
6. **Verify npm:** Check packages at npmjs.com/@bradygaster/squad-cli and squad-sdk
137+
7. **Post-release bump:** Bump dev branch to {NEXT}-preview.1
138+
139+
## Rollback Procedures
140+
141+
**If semver invalid:**
142+
1. Delete tag: `git tag -d v{VERSION} && git push origin :refs/tags/v{VERSION}`
143+
2. Revert commit: `git revert {commit}`
144+
3. Fix version and retry
145+
146+
**If npm publish fails:**
147+
1. Check workflow logs for error
148+
2. Fix error (token, version, etc.)
149+
3. Re-trigger: `gh workflow run publish.yml --ref v{VERSION}`
150+
151+
**If wrong version published:**
152+
1. Within 72 hours: `npm unpublish @bradygaster/squad-cli@{VERSION}`
153+
2. After 72 hours: Publish corrected version with patch bump
154+
155+
## Known Failure Modes
156+
157+
See `.squad/agents/kobayashi/charter.md` Failure 3 for complete incident report.
158+
```
159+
160+
**Benefits:**
161+
- Single source of truth for release process
162+
- Includes pre-flight checklist
163+
- Documents rollback procedures
164+
- Can be loaded on-demand by coordinator
165+
166+
## Implementation Priority
167+
168+
**High priority (implement now):**
169+
1. ✅ Pre-publish semver validation (5 min, zero risk)
170+
2. ✅ GitHub Release draft verification (10 min, self-correcting)
171+
172+
**Medium priority (implement before next release):**
173+
3. ⚠️ NPM_TOKEN type verification (15 min, requires dry-run testing)
174+
175+
**Low priority (nice-to-have):**
176+
4. 📝 Release runbook skill (30 min, documentation effort)
177+
178+
## Backward Compatibility
179+
180+
**Zero breaking changes:**
181+
- All changes are additive (new validation steps)
182+
- Existing valid releases will pass all checks
183+
- Invalid releases will now fail fast (intended behavior)
184+
185+
## Testing Strategy
186+
187+
**Validation steps:**
188+
1. Test with valid semver: 0.8.22 → should pass
189+
2. Test with 4-part version: 0.8.21.4 → should FAIL with clear error
190+
3. Test with version mismatch: tag=0.8.22, package.json=0.8.21 → should FAIL
191+
4. Test with draft release → should auto-publish or fail with actionable message
192+
193+
**NPM token test:**
194+
1. Create test automation token on npmjs.com
195+
2. Configure in repo secrets
196+
3. Run dry-run publish → should pass
197+
4. Switch to user token with 2FA → should FAIL with EOTP error message
198+
199+
## Success Metrics
200+
201+
**Before:**
202+
- v0.8.22 incident: 4+ failures, multiple Brady interventions, hours to resolve
203+
204+
**After:**
205+
- Invalid semver caught in CI before reaching npm
206+
- Draft releases auto-corrected or blocked
207+
- Token issues detected before first publish attempt
208+
- Release process completes in <10 minutes with zero manual intervention
209+
210+
## Decision Request
211+
212+
**Approve these guardrails for immediate implementation?**
213+
214+
- [ ] Approve all (implement now)
215+
- [ ] Approve high-priority only (defer medium/low)
216+
- [ ] Request changes (specify below)
217+
218+
**Brady's decision:**

0 commit comments

Comments
 (0)