Feature/rag evaluation - #1243
Conversation
… as zscaler uses 9000 atm
![datadog-uktrade[bot]](https://avatars.githubusercontent.com/in/1041283?s=60&v=4){kind=small}
| try: | ||
| s3.delete_object(Bucket=env.bucket_name, Key=key) | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
🔴 High: Code Quality Violation
silent exception (...read more)
Using the pass statement in an exception block ignores the exception. Exceptions should never be ignored. Instead, the user must add code to notify an exception occurred and attempt to handle it or recover from it.
The exception to this rule is the use of StopIteration or StopAsyncIteration when implementing a custom iterator (as those errors are used to acknowledge the end of a successful iteration).
Helpful? 👍/👎 
🚩 Report as false positive. Flags the rule for review to improve detection accuracy.
| # HTML | ||
| # ------------------------------------------------------------------ | ||
|
|
||
| def _build_html(self, agg: AggregateScores, meta: dict) -> str: |
There was a problem hiding this comment.
🔴 High: Code Quality Violation
function exceeds 200 lines (...read more)
This rule stipulates that functions in Python should not exceed 200 lines of code. The primary reason for this rule is to promote readability and maintainability of the code. When functions are concise and focused, they are easier to understand, test, and debug.
Long functions often indicate that a single function is doing too much. Adhering to the Single Responsibility Principle (SRP) can help avoid this. SRP states that a function should have only one reason to change. If a function is doing more than one thing, it can usually be split into several smaller, more specific functions.
In practice, to adhere to this rule, you can often break up long functions into smaller helper functions. If a piece of code within a function is independent and can be isolated, it is a good candidate to be moved into a separate function. This also increases code reusability. For instance, if a function process_data() is too long, you can identify independent tasks within it - such as clean_data(), transform_data(), and save_data() - and create separate functions for them. This makes the code easier to reason about and test, and promotes good coding practices.
Helpful? 👍/👎
🚩 Report as false positive. Flags the rule for review to improve detection accuracy.
|
Dependency Review✅ No vulnerabilities or OpenSSF Scorecard issues found.Scanned FilesNone |
|
Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run |
|
Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #1243 +/- ##
==========================================
- Coverage 85.54% 85.53% -0.02%
==========================================
Files 231 231
Lines 10406 10409 +3
==========================================
+ Hits 8902 8903 +1
- Misses 1504 1506 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run |
|
Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run |
|
Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run |
|
Your PR has commits that are missing the Signed-off-by trailer. This is likely due to the pre-commit hook not being configured on your local machine. The usual fix for this issue is to run |
2 participants
Context
Assist does not have an existing RAG evaluation pipeline. This change produce a basic framework that is extensible to accomodate further ideas
What
It creates a pipeline that leverage Assist production instances to evaluate the retrieval quality of the application. It will iterate over a number of pdfs in corpus, exract, chunk, embed and index them in a test index within open search. It then iterates ove QA in corpus, to send the question to Assist retriever instance and compute metrics based on retrieved chunks and the corresponding answers in QA.
Have you written unit tests?
Are there any specific instructions on how to test this change?
Instructions are available in the README
Relevant links