Fix RCA agent confabulation via prompt restructuring - #550
Conversation
The agent was fabricating root causes because background_source_general.md forced "Identify root cause with evidence" as a mandatory step, overriding the conclusion_gate.md guidance that inconclusive is acceptable. This made the agent construct plausible narratives connecting real events to unconfirmed symptoms. Changes: - background_source_general.md: Step 5 now allows confirmed, likely, or inconclusive outcomes (depth requirement of 15-20 tool calls unchanged) - conclusion_gate.md: Add classification task (CONFIRMED/LIKELY/INCONCLUSIVE) that must be resolved before narrative generation, with explicit guard that INCONCLUSIVE requires exhausting investigation first - evidence_standard.md: Reinforce that inconclusive is a valid outcome and confidence must match evidence
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughThe PR aligns RCA guidance and summarization prompts around confirmed, likely, and inconclusive conclusions, adds self-check and evidence-handling rules, and updates transcript-fallback wording to match the same certainty and anti-hallucination contract. ChangesRCA certainty guidance
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![arvo-ai-staging[bot]](https://avatars.githubusercontent.com/in/4088260?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@server/chat/background/summarization.py`:
- Line 332: The summary text in summarization should use a plain hyphen instead
of an EN DASH in the “2-3 paragraph” range to match the rest of the file and
satisfy the lint rule. Update the prompt text in the affected summarization
content so the range uses HYPHEN-MINUS consistently, and keep the rest of the
instructions unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: bd0c0cc9-ed0e-4951-b438-ad78bbe24225
📒 Files selected for processing (4)
server/chat/backend/agent/prompt/rca_sections/conclusion_gate.mdserver/chat/backend/agent/prompt/rca_sections/evidence_standard.mdserver/chat/backend/agent/skills/rca/background/background_source_general.mdserver/chat/background/summarization.py
1ff8c2b
|
2 participants
The agent was fabricating root causes because background_source_general.md forced "Identify root cause with evidence" as a mandatory step, overriding the conclusion_gate.md guidance that inconclusive is acceptable. This made the agent construct plausible narratives connecting real events to unconfirmed symptoms.
Changes:
Summary by CodeRabbit