Skip to content
Closed
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
1595ba8
Add Baseline + Graph RAG tools
gitbuda Aug 29, 2025
0d155c9
Add working LightRAG example
gitbuda Aug 29, 2025
78e79b2
Add evaluations
gitbuda Aug 30, 2025
197bdfe
Add information density dummy metric
gitbuda Aug 30, 2025
1e0cff3
Add unstructured test data
gitbuda Aug 31, 2025
135a0cf
Merge branch 'add-basic-rag-evaluations' into add-baseline-graph-rag-…
gitbuda Aug 31, 2025
d90467a
Add the uv structure
gitbuda Aug 31, 2025
c5138bf
Merge branch 'add-basic-rag-evaluations' into add-baseline-graph-rag-…
gitbuda Aug 31, 2025
45f716a
Add print evaluation summary
gitbuda Aug 31, 2025
88cd186
Merge branch 'add-basic-rag-evaluations' into add-baseline-graph-rag-…
gitbuda Aug 31, 2025
deea282
Run dummy evaluations on the kagglehub simple dataset
gitbuda Aug 31, 2025
5edd650
Merge branch 'main' into add-baseline-graph-rag-tools
gitbuda Sep 6, 2025
af2ec72
Add the new tools to the mcp server
gitbuda Sep 6, 2025
bfc9a15
Update the project version
gitbuda Sep 6, 2025
46fb5d9
Merge branch 'add-new-tools-to-the-mcp-server' into add-baseline-grap…
gitbuda Sep 6, 2025
0a88c9e
Add an attempt to make litellm + mcp client prompt lib (WIP/not_working)
gitbuda Sep 6, 2025
f7cd46d
Add the working version
gitbuda Sep 7, 2025
08cbc40
Make the StdioServerParams dynamically resolved
gitbuda Sep 7, 2025
4aeebe3
Add prompt-evals and mcp-prompt-lib as workspaces
gitbuda Sep 7, 2025
921aab1
Add evals to prompt evals
gitbuda Sep 7, 2025
c55e88f
Improve a bit on the calling MCP tools side
gitbuda Sep 8, 2025
244c516
Add the TODO parameters to run under hosted_vllm/openai/gpt-oss-20b
gitbuda Sep 8, 2025
9546a2e
Replace evaluations/ with DeepEval
gitbuda Sep 9, 2025
e6950c9
Add llm_model_resolver
gitbuda Sep 9, 2025
b757d9a
Fix the logger
gitbuda Sep 9, 2025
aa34f4f
Polish the prompt-evals CLI params
gitbuda Sep 10, 2025
34ebb10
Add more stuff to the model resolver because we are using different m…
gitbuda Sep 10, 2025
4556855
Add the hint prompt
gitbuda Sep 10, 2025
f8715d2
Resolve the asyncio calling issue (.run shoule be called once per pro…
gitbuda Sep 10, 2025
5ee0693
Add sentence embedding util
gitbuda Sep 13, 2025
144458a
Merge branch 'add-sentence-embedding-util' into add-baseline-graph-ra…
gitbuda Sep 14, 2025
e832af8
Try the prompt lib on another use-case
gitbuda Sep 14, 2025
e3c1693
Improve the prompt-lib in a way that MCP server params can be injected
gitbuda Sep 14, 2025
84ef68a
Fix the NodeVectorSearch tool call
gitbuda Sep 15, 2025
cc94d8a
Add experiments/entity-graph
gitbuda Sep 15, 2025
d048c59
Add lightrag to experiments
gitbuda Sep 16, 2025
45d16b1
Add configurable logging level to the memgraph-toolbox -> A NEW PR
gitbuda Sep 17, 2025
afe87b8
Add custom DeepEval Coherence metric
gitbuda Sep 17, 2025
7099300
Add logger to the mcp source file
gitbuda Sep 18, 2025
9ee6081
Try to plugin the HB model directly into LightRAG
gitbuda Sep 18, 2025
07b119a
Comment out the HB model
gitbuda Sep 18, 2025
8fd9698
Add MCP client prompt lib
gitbuda Sep 19, 2025
71acd5e
Add coherence embeddings-based DeepEval metric
gitbuda Sep 19, 2025
1ea365c
Remove parts that were move to a separated PRs
gitbuda Sep 19, 2025
679d94d
Merge the legit prompt lib
gitbuda Sep 19, 2025
795e0c1
Merge the legit coherence eval
gitbuda Sep 19, 2025
b280620
Remove the bitsandbytesbecause id doesn't work on mac
gitbuda Sep 19, 2025
1d55450
Remove the flash attention
gitbuda Sep 19, 2025
6657e5f
Polish and add tests
gitbuda Sep 20, 2025
2d98302
Fix the failing CI
gitbuda Sep 20, 2025
47dda21
Merge the embedding util
gitbuda Sep 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.10
3.13
1 change: 1 addition & 0 deletions evaluations/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Knowledge Retrieval Evaluation Tools
39 changes: 39 additions & 0 deletions evaluations/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""
Knowledge Retrieval Evaluation Tools

This package provides functionality for evaluating knowledge retrieval
responses in different scenarios. The aim is to design tools to be LLM agnostic
and easily extensible.
"""

from .main import (
TestCase,
TestRun,
EvaluationResult,
EvaluationScore,
BaseMetric,
AnswerRelevancyMetric,
FaithfulnessMetric,
ContextualPrecisionMetric,
InformationDensityMetric,
Evaluator,
evaluate,
print_evaluation_summary,
)

__version__ = "0.1.0"

__all__ = [
"TestCase",
"TestRun",
"EvaluationResult",
"EvaluationScore",
"BaseMetric",
"AnswerRelevancyMetric",
"FaithfulnessMetric",
"ContextualPrecisionMetric",
"InformationDensityMetric",
"Evaluator",
"evaluate",
"print_evaluation_summary",
]
Loading
Loading