Fix RCA agent confabulation via prompt restructuring

server/chat/backend/agent/prompt/rca_sections/conclusion_gate.md

@@ -7,8 +7,32 @@ You will conclude too early. Recognize these traps:
 - "The service restarted, so resource exhaustion": check actual resource metrics.
 - "We need to scale up resources": that's a band-aid, not a root cause. Why are resources insufficient now? Did something change or was it always underprovisioned?
 - "The cluster is unstable": what specifically is making it unstable? Which node, which component, what changed?
+- "I found a change that could cause this": could is not did. Where is the runtime evidence it actually happened?
+- "I found errors related to the reported symptom": errors existing is not the same as users being impacted. Server Action errors during deploys, connection resets during pod cycling, and timeout spikes during scaling are normal operational noise. Confirm the symptom is CURRENTLY affecting users, not just that related errors exist in logs.
+
+Absence of expected evidence is evidence. If you searched for error logs matching the reported symptom and found none, that is not a gap in your investigation — it is a finding. It means the symptom may not be occurring, or your hypothesis is wrong. Do not construct a theoretical explanation for why errors SHOULD exist when you cannot find them.
+
+# Classify Before Presenting
+
+After completing your investigation (steps 1-4, 15-20+ tool calls), classify your conclusion strength BEFORE writing it up:
+
+- CONFIRMED: You verified the reported symptom is actively occurring AND traced it to a specific cause with evidence at each link. "The reported symptom" is whatever the alert is about — for user-facing services that means user-impacting signals (e.g., login endpoint returning errors NOW, users reporting failures, health check failing at investigation time); for internal/non-user-facing systems it means the failing job, stalled pipeline, breached SLO, or affected dependency itself (e.g., the batch run actually failing, replication lag exceeding threshold NOW). Finding related errors that could theoretically cause the symptom is not confirmation — you must show the symptom itself is manifesting.
+- LIKELY: You found a plausible cause but could not directly observe the reported symptom in runtime data. Present as hypothesis, not fact.
+- INCONCLUSIVE: You could not confirm the reported symptom is occurring, or multiple equally-plausible causes exist with no differentiating evidence. Present what was ruled out.
+
+If you cannot classify as CONFIRMED, do not present your finding as a definitive root cause. A LIKELY finding is a hypothesis. An INCONCLUSIVE finding reports what was investigated and eliminated.
+
+You may only reach INCONCLUSIVE after exhausting your investigation — not as a shortcut. If you have unchecked data sources, you are not done investigating.
+
+# Self-Check
 
 Before stating root cause, answer:
 1. What alternative did you rule out, and how?
-2. What specific evidence (tool output) proves the mechanism, not just the correlation?
+2. What specific evidence shows the reported symptom is CURRENTLY occurring — not just that related errors exist? A Server Action error in logs does not confirm login is broken. A timeout in Gunicorn does not confirm requests are failing. What shows the symptom itself is manifesting RIGHT NOW? Point to direct evidence of the reported problem (for user-facing services: error rates on the endpoint, failed health checks, 5xx responses to clients; for internal systems: the failed/stalled job, the breached metric, the affected downstream). If you cannot point to such evidence, your classification is LIKELY at best.
 3. Does your root cause explain the timing of the alert?
+
+# When Evidence Is Insufficient
+
+"Insufficient evidence to determine root cause" is a correct and complete answer when the evidence does not clearly support one. Stating what you confirmed, what you ruled out, and what remains unknown is more valuable than a confident guess. A wrong root cause wastes engineering time; an honest "unclear" focuses investigation where it's needed.
+
+Do not invent a root cause to fill the gap. If you have a leading hypothesis but cannot confirm the mechanism, present it explicitly as unconfirmed: "Most likely X based on [evidence], but could not confirm because [what's missing]."

server/chat/backend/agent/prompt/rca_sections/evidence_standard.md

@@ -2,6 +2,6 @@
 
 Never state a root cause without citing the specific tool output that proves it. Exact timestamps, error messages, metric values.
 
-If you cannot determine root cause, say what you confirmed, what you ruled out, and what remains unverified. Distinguish facts from hypotheses in your reporting.
+If you cannot determine root cause, say what you confirmed, what you ruled out, and what remains unverified. This is a successful investigation outcome — not a failure. Distinguish facts from hypotheses in your reporting.
 
-Do not fabricate log lines, metrics, or timestamps. Do not hedge confirmed findings or overclaim uncertain ones. Match your confidence to the evidence.
+Do not fabricate log lines, metrics, or timestamps. Do not hedge confirmed findings or overclaim uncertain ones. Match your confidence to the evidence. A finding stated at the wrong confidence level is a lie regardless of direction.

server/chat/backend/agent/skills/rca/background/background_source_general.md

@@ -3,7 +3,7 @@ MANDATORY INVESTIGATION STEPS - DO NOT STOP UNTIL ALL ARE DONE:
 2. If SSH keys are already available in ~/.ssh/, SSH into an affected VM for system-level diagnostics. If SSH is unavailable or access is denied, use cloud provider APIs and monitoring tools instead - do NOT attempt to generate keys or bypass access controls.
 3. Check system metrics via available tools (cloud monitoring APIs, kubectl top, or SSH if accessible)
 4. Check logs: cloud logging, kubectl logs, monitoring integrations, or SSH if accessible
-5. Identify root cause with evidence
+5. State your conclusion based on evidence gathered above: confirmed root cause, likely hypothesis, or what was ruled out if inconclusive
 6. Provide remediation steps
 
 YOU MUST make 15-20+ tool calls. After EACH tool call, continue investigating.

server/chat/background/summarization.py

@@ -279,9 +279,11 @@ def _build_summary_prompt_with_chat(
 State what occurred, when, and what was affected, using only facts present in the provided sections. Report as known facts, not as an investigation log.
 
 PARAGRAPH 2 — Root Cause:
-- If the numbered evidence clearly supports a specific root cause, state it and cite the supporting evidence with numeric markers like [3], [7].
+- If the investigator notes classify the finding as CONFIRMED: state the root cause confidently and cite the supporting evidence with numeric markers like [3], [7].
   Example: "The root cause was a ConfigMap change that increased BATCH_SIZE from 1000 to 10000 [7], causing memory usage to exceed the 128Mi limit [9, 11]."
-- Otherwise the paragraph MUST begin with "Root cause undetermined." and describe what is and isn't known. Do not invent a cause to fill the paragraph.
+- If the investigator notes classify the finding as LIKELY: begin with "The most likely cause is..." or "Evidence suggests..." and present as hypothesis, NOT as confirmed fact. Include what could not be verified.
+  Example: "The most likely cause is a stale client bundle sending requests to deprecated Server Action IDs after the 12:59 UTC deployment [7, 11], though direct confirmation that users are currently unable to log in was not obtained during the investigation."
+- If the investigator notes classify the finding as INCONCLUSIVE, or indicate the investigation was inconclusive: the paragraph MUST begin with "Root cause undetermined." and describe what is and isn't known. Do not invent a cause to fill the paragraph.
 
 PARAGRAPH 3 (if significant) — Impact & Timeline:
 Only the scope and timeline actually present in the alert or evidence.
@@ -292,6 +294,7 @@ def _build_summary_prompt_with_chat(
 - Never fabricate a [n] citation — every marker MUST reference a real numbered evidence row above.
 - Treat purely temporal correlations as correlations, not causations.
 - If the investigator notes indicate the investigation was inconclusive, the Root Cause paragraph MUST start with "Root cause undetermined."
+- If the investigator notes classify the finding as LIKELY, do NOT state the root cause as confirmed fact. Use hedged language ("most likely", "evidence suggests") and note what remains unverified.
 
 CITATION RULES (non-negotiable):
 - ONLY cite numbered evidence items using their numeric index: [1], [3, 5], [7].
@@ -326,13 +329,14 @@ def _build_summary_prompt_with_chat(
 INVESTIGATION TRANSCRIPT (chat log):
 {transcript}
 
-Write a concise 2–3 paragraph summary covering what triggered the alert, the severity and observed impact (only if explicitly present), the affected service, and the best-known root cause. If the root cause is not explicit, the Root Cause paragraph MUST begin with "Root cause undetermined." and describe what is and isn't known.
+Write a concise 2–3 paragraph summary covering what triggered the alert, the severity and observed impact (only if explicitly present), the affected service, and the best-known root cause. If the root cause is not explicit or the investigator notes classify it as INCONCLUSIVE, the Root Cause paragraph MUST begin with "Root cause undetermined." and describe what is and isn't known. If classified as LIKELY, use hedged language ("most likely", "evidence suggests") and note what remains unverified.
 
 ANTI-HALLUCINATION RULES (non-negotiable):
 - Use only facts present in the provided sections — do not introduce services, hosts, configs, error messages, or metrics that aren't there.
 - Empty or missing output from a generic probe is NOT evidence of a specific failure mode.
 - Treat purely temporal correlations as correlations, not causations.
 - If the investigator notes indicate the investigation was inconclusive, the Root Cause paragraph MUST start with "Root cause undetermined."
+- If the investigator notes classify the finding as LIKELY, do NOT state the root cause as confirmed fact.
 
 Tone: neutral, factual, incident-record style. Descriptive, not advisory. Do not address any audience.