Evaluate Metric Stability Across Different LLMs

[cyx96]

Summary

Run a controlled experiment to measure how open-rag-eval metrics change when using different LLMs to generate answers. This helps us understand whether certain metrics are stable, noisy, or biased depending on the model family.

---

Task Description

Using a fixed retrieval pipeline (same dataset, retriever, and retrieved passages), evaluate RAG answers generated by multiple LLMs such as:

• GPT-5.x
• Claude 4.x
• Gemini 3.x
• DeepSeek-V3.x / DeepSeek-R1
• Llama-4
• or similar open-weights

For each LLM, run open-rag-eval on the same query set and compare metric performance.

---

What to Produce

• A table comparing metrics across models (e.g., Recall@K, Faithfulness, Nugget Precision, Hallucination rate).
• 1–2 visualizations (bar chart or heatmap).
• A short interpretation of which metrics vary significantly across models and why.

---

Acceptance Criteria

• Fixed dataset + retrieval configuration
• Reproducible evaluation config checked in (YAML or CLI commands)
• At least 3 LLMs evaluated
• All metrics compared
• Results table included
• Visualization included

---

Notes

• Focus on metric behavior, not model performance.
• This issue helps us understand whether some metrics are overly sensitive to LLM generation style or length.