[Laurent Nguyen (languy)]

• Added detailed summaries for unresolved failures, including inline evidence and links to failed steps.
• Improved error handling and logging for evaluation attempts.
• Introduced new functions for summarizing evaluation results and extracting error messages.
• Updated tests to cover new summary features and ensure accurate reporting of evaluation outcomes.

Description

Improve summary page of the eval:

• Remove the successful runs table: instead we display the count of successful runs. If tokens, elapsed time is interesting, then a link to download the artifact is now displayed.
• Only display errors with link to the failed step. Example here.
• Note that the "Failed Stimulus" column contains hyperlinks to the tests

Type of Change

• 📝 New rule - Adding a new best practice rule
• ✏️ Rule improvement - Updating an existing rule
• 🆕 New skill - Adding an entirely new skill
• 🐛 Bug fix - Fixing an issue with existing content
• 📚 Documentation - Updating README, CONTRIBUTING, or other docs
• 🔧 Build/Scripts - Changes to build process or scripts

Checklist

• I have read the Contributing Guide
• I ran npm run validate and it passed
• My rule file follows the naming convention: {prefix}-{description}.md
• My rule includes valid frontmatter (title, impact, tags)

For New Rules

Rule file: skills/cosmosdb-best-practices/rules/_____.md

Category:

Impact level:

Why is this rule important?

Agent Testing

• Tested with GitHub Copilot
• Tested with Claude Code
• Tested with Cursor
• Tested with other agent: _____
• N/A (documentation only)

Related Issues

Additional Notes

[Copilot]

🟡 Changes recommended

An uncaught parse error on a truncated JSONL record prevents the publisher from reporting persisted termination errors and writing either summary output.

1 open issue

• Handle truncated JSONL records without skipping termination reporting · New

What changed in this PR

Improves Vally evaluation reporting with failure-focused GitHub Actions summaries while keeping detailed results available in downloadable artifacts.

Changes:

• Shows unresolved failures with compact evidence and links to test definitions and pipeline steps.
• Records evaluation attempts and process errors to support summary generation.
• Adds summary tests and documents the reporting behavior.

File	Description
scripts/​eval.mjs	Persists attempt status, errors, and result-file paths.
scripts/​eval-summary.test.mjs	Tests summaries, error handling, source links, and artifact integration.
scripts/​eval-summary.mjs	Generates failure-focused summaries from recorded attempts.
scripts/​eval-retry.mjs	Exposes threshold-aware failed-trial selection.
README.md	Documents summaries and downloadable results.
package.json	Includes summary tests in the evaluation test command.
.github/​workflows/​evals.yml	Captures logs, publishes summaries, and links artifacts and failed steps.

_{Review effort: Balanced}