Hi,
I maintain WFGY ProblemMap, a 16-mode checklist for common failure modes in RAG and agentic systems:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md
The same checklist is already referenced in:
- ToolUniverse (Harvard MIMS Lab)
- Rankify (Univ. of Innsbruck Data Science Group)
- The Multimodal RAG Survey (QCRI LLM Lab)
open-rag-eval sits exactly at the place where these failure modes show up: multi-step reasoning, agent coordination, and RAG-style tool use.
Proposal (docs only, low maintenance):
- Add a Failure Modes Checklist:
- reuse the 16 ProblemMap categories, but illustrated with open-rag-eval style workflows
- for each mode, list 1–2 signals users can log (e.g. inconsistent tool traces, context reuse patterns, retrieval-generation mismatch)
- Optionally add a short guide: “If your open-rag-eval agent behaves strangely, classify the failure here first.”
If you like this, I can open a PR with a first draft based on the existing ProblemMap, adapted to open-rag-eval ’s architecture.
Happy to adjust to your preferred doc layout and terminology.
Hi,
I maintain WFGY ProblemMap, a 16-mode checklist for common failure modes in RAG and agentic systems:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md
The same checklist is already referenced in:
open-rag-eval sits exactly at the place where these failure modes show up: multi-step reasoning, agent coordination, and RAG-style tool use.
Proposal (docs only, low maintenance):
If you like this, I can open a PR with a first draft based on the existing ProblemMap, adapted to open-rag-eval ’s architecture.
Happy to adjust to your preferred doc layout and terminology.