Hi open-rag-eval maintainers — I run EvalPort, an open, portable schema for LLM/RAG eval results (openeval.validate.validate_suite() / validate_result_set()). The idea is a small tool-agnostic JSON shape so results produced by one eval framework can be read, diffed, or re-scored by another instead of everyone inventing their own CSV/JSON layout.
I read open_rag_eval/data_classes/rag_results.py and eval_scores.py before writing this, since a bare metric-returns-a-float wouldn't be a good fit:
RAGResult (retrieval_result: RetrievalResult, generation_result: AugmentedGenerationResult) and MultiRAGResult (query, query_id, rag_results: List[RAGResult], optional expected_answer) are real per-query result objects, not just numbers.RAGScores (retrieval_score: RetrievalScores, generation_score: AugmentedGenerationScores) and ScoredRAGResult (rag_result: RAGResult, scores: RAGScores) pair those results with per-metric score dicts, and TRECEvaluator/ConsistencyEvaluator (in evaluators/) emit these per-query, writing the aggregate to results.json/CSV.
That per-query structure — query, retrieved passages, cited generation, and a named-metric score dict — maps well onto an OpenEval ResultSet item (one graded record per example, with named scores attached).
What I'm proposing: a small adapter (could live under open_rag_eval/ next to the existing LangChain/LlamaIndex/ChromaDB connectors, or in EvalPort itself) that converts a list of ScoredRAGResult/MultiScoredRAGResult into an OpenEval ResultSet passing validate_result_set() — and ideally the reverse, so an EvalPort-format test suite could drive open_rag_eval's evaluators. Happy to put up a first-draft PR against either repo, or just help spec the field mapping if someone here wants to build it. Low-pressure either way — mostly flagging the idea now that I've actually looked at the data model.
Repo/spec: https://github.com/adhabnr-ux/evalport
Hi open-rag-eval maintainers — I run EvalPort, an open, portable schema for LLM/RAG eval results (
openeval.validate.validate_suite()/validate_result_set()). The idea is a small tool-agnostic JSON shape so results produced by one eval framework can be read, diffed, or re-scored by another instead of everyone inventing their own CSV/JSON layout.I read
open_rag_eval/data_classes/rag_results.pyandeval_scores.pybefore writing this, since a bare metric-returns-a-float wouldn't be a good fit:RAGResult(retrieval_result: RetrievalResult,generation_result: AugmentedGenerationResult) andMultiRAGResult(query,query_id,rag_results: List[RAGResult], optionalexpected_answer) are real per-query result objects, not just numbers.RAGScores(retrieval_score: RetrievalScores,generation_score: AugmentedGenerationScores) andScoredRAGResult(rag_result: RAGResult,scores: RAGScores) pair those results with per-metric score dicts, andTRECEvaluator/ConsistencyEvaluator(inevaluators/) emit these per-query, writing the aggregate toresults.json/CSV.That per-query structure — query, retrieved passages, cited generation, and a named-metric score dict — maps well onto an OpenEval
ResultSetitem (one graded record per example, with named scores attached).What I'm proposing: a small adapter (could live under
open_rag_eval/next to the existing LangChain/LlamaIndex/ChromaDB connectors, or in EvalPort itself) that converts a list ofScoredRAGResult/MultiScoredRAGResultinto an OpenEvalResultSetpassingvalidate_result_set()— and ideally the reverse, so an EvalPort-format test suite could driveopen_rag_eval's evaluators. Happy to put up a first-draft PR against either repo, or just help spec the field mapping if someone here wants to build it. Low-pressure either way — mostly flagging the idea now that I've actually looked at the data model.Repo/spec: https://github.com/adhabnr-ux/evalport