GIE-507: [release-0.3] update observability task prompts to request metric names and PromQL queries#276
Conversation
…PromQL queries The observability eval tasks were failing because agents correctly answered questions but summarized results in natural language without restating the Prometheus metric name. The llmJudge contains assertions check the agent's response text, not tool call arguments, so correct answers were marked as failures. Update 15 task prompts to explicitly ask the agent to include the metric name and PromQL query used. This ensures the metric name appears in the response for the judge to verify, while keeping the contains assertions strict and non-overlapping with prompt text. Signed-off-by: Jayapriya Pai <janantha@redhat.com>
|
@openshift-cherrypick-robot: Ignoring requests to cherry-pick non-bug issues: GIE-507 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@openshift-cherrypick-robot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
@Cali0707 for backport approval |
|
/override "Red Hat Konflux / openshift-mcp-server-release-03-on-pull-request" |
|
@matzew: Overrode contexts on behalf of matzew: Red Hat Konflux / openshift-mcp-server-release-03-on-pull-request DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: matzew, openshift-cherrypick-robot The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/retitle GIE-507: [release-0.3] update observability task prompts to request metric names and PromQL queries |
openshift-ci
Bot
changed the title
|
/jira refresh |
openshift-ci-robot
added
the
jira/valid-reference
|
@slashpai: This pull request references GIE-507 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-merge-bot
Bot
merged commit de5dfc5
into
openshift:release-0.3
4 participants
This is an automated cherry-pick of #274
/assign slashpai