CI Failure DoctorCI Failure Investigation: Daily Test Improver - Persistent "Setup threat detection" Failure (30+ Days)

[github-actions[bot]]

Summary

The Daily Test Improver workflow has been consistently failing at the "Setup threat detection" step in the detection job since at least March 1, 2026 — over 29 consecutive daily failures. The same failure affects Daily Perf Improver (every run) and intermittently Daily QA (when the agent produces output).

---

Failure Details

• Run: #167 (23705085085)
• Commit: 3fa0e5d105b8ebfa8802002edc5fb117ebb237bf
• Trigger: schedule (daily)
• Workflow: Daily Test Improver

---

Root Cause Analysis

This is not a code issue in the rlm-rs codebase. The fastembed bump from 5.12.1 → 5.13.0 (PR #159) is not the cause — failures predate that commit by weeks.

The failure is in the gh-aw agentic workflow infrastructure, specifically the setup_threat_detection.cjs script shipped with github/gh-aw/actions v0.47.1 (pinned at commit 1b847e3d0c6d8ebc44cb538c55198da42baa8a78 in all .lock.yml files).

Failure path:

1. activation job → ✅ success
2. agent job → ✅ success (the AI agent actually runs correctly every time)
3. detection job → ❌ failure at Step 6: Setup threat detection (fails within <1 second)
4. safe_outputs → ⏭ skipped (because detection failed)
5. conclusion → ✅ success (graceful degradation: "Handle Agent Failure")

The setup_threat_detection.cjs script fails when invoked after the agent produces output (HAS_PATCH=true). This explains why:

• Daily Test Improver: fails 100% of runs (agent always writes tests → always has patches)
• Daily Perf Improver: fails 100% of runs (agent always generates improvements)
• Daily QA: fails intermittently — only when agent produces output, succeeds on no-op days

---

Failed Jobs and Errors

Job	Step	Conclusion
detection	Setup threat detection (step 6)	❌ failure
detection	Validate COPILOT_GITHUB_TOKEN secret	⏭ skipped
detection	Execute GitHub Copilot CLI	⏭ skipped
safe_outputs	(entire job)	⏭ skipped

---

Investigation Findings

Pattern timeline:

• ✅ Last successful Daily Test Improver run: Feb 28, 2026 (run [agentics] No-Op Runs #33)
• ❌ Continuous failures began: March 1, 2026
• Total consecutive daily failures: 29+ days

Version mismatch identified:

• Lock files pin github/gh-aw/actions/setup at v0.47.1 (commit 1b847e3d0c6d8ebc44cb538c55198da42baa8a78)
• Open PR ci: bump github/gh-aw from 0.58.3 to 0.62.5 #161 proposes bumping github/gh-aw from 0.58.3 to 0.62.5
• The agentics-maintenance.yml already references v0.58.3 — only the lock files are stale

Key diagnostic fact: The agent runs successfully every time. The AI is doing its job (writing tests, making improvements). The issue is purely in the post-agent security validation pipeline. No agent output ever reaches safe_outputs due to the detection failure.

Related issues:

• Issue [agentics] Daily Test Improver failed #77 "[agentics] Daily Test Improver failed" — created March 1, closed March 6 (without fixing root cause)
• Issue [aw] CI Failure Doctor failed #124 "[aw] CI Failure Doctor failed" — the CI Doctor was also failing for related reasons

---

Recommended Actions

• Merge PR ci: bump github/gh-aw from 0.58.3 to 0.62.5 #161 to update github/gh-aw from 0.58.3 to 0.62.5
• Recompile workflow lock files by running gh aw compile in the repository root — this regenerates all .lock.yml files from their .md sources using the latest gh-aw version
• Verify: After recompile, check that setup_threat_detection.cjs version bundled in the new gh-aw release is compatible with current agent output format
• Monitor: Confirm next daily run of Daily Test Improver, Daily Perf Improver, and Daily QA all pass the detection job

---

Prevention Strategies

1. Enable dependabot for gh-aw actions: Ensure PR ci: bump github/gh-aw from 0.58.3 to 0.62.5 #161 (and similar bump PRs) are merged promptly — the 30-day gap between gh-aw version and lock file version is the root issue
2. Add detection job monitoring: Add an alert or issue if the detection job fails 3+ consecutive times
3. Lock file freshness check: Add a CI check that validates the lock files were compiled with the current gh-aw version

---

AI Team Self-Improvement

For AI coding agents working on this repository, add to AGENTS.md or instructions.md:

## Agentic Workflow Maintenance

When the Daily Test Improver, Daily Perf Improver, or Daily QA workflows fail at the
"Setup threat detection" step in the detection job, this indicates a stale gh-aw lock file.
The fix is NOT in the Rust code - it requires running `gh aw compile` to regenerate the
.lock.yml files in .github/workflows/ from their .md sources. Check if there is a pending
dependabot PR for `github/gh-aw` version bump and merge it first.

---

Historical Context

• Issue [agentics] Daily Test Improver failed #77 was the first recorded Daily Test Improver failure (March 1, 2026), closed without permanent fix
• Issue [aw] CI Failure Doctor failed #124 tracked the CI Failure Doctor itself failing (separate but related gh-aw infrastructure issue)
• This failure pattern has persisted for 29+ days with no code-level resolution
• The actual Rust code and tests are unaffected — the agent successfully analyzes the repo, but its outputs (test improvements, PRs) are silently discarded due to the detection failure

AI generated by CI Failure Doctor · history

• expires on Apr 5, 2026, 8:48 AM UTC