skills-evaluator

Lightweight evaluator and optimizer for agent skills, built around the core question:

Does this skill add meaningful value beyond what the base model already knows?

The system implements the matrix from docs/2026-02-17-plan.md and docs/2026-02-17-context.md with an explicit split between:

• selection effects (routing quality)
• execution effects (content value vs sham)
• end-to-end effects (real net gain with cost/latency)

Why this design

This project is intentionally minimal and practical for expensive agent runs.

• Uses a fixed condition matrix: baseline, selector, oracle_real, oracle_sham, forced_negative, auto_real
• Uses sham/placebo control to separate meaningful content from token-load effects
• Produces a strict verdict: helpful, helpful_routing_limited, neutral, harmful
• Optimizes in phases (metadata first, then body) with local mutations by default and optional GEPA backend

Tooling

• mise for tool/task orchestration
• uv for Python env/dependencies
• ruff for lint/format

mise run setup
mise run lint
mise run test

Project layout

• src/skills_evaluator/models.py: schemas for skills, tasks, rows, metrics, verdicts
• src/skills_evaluator/skill_artifact.py: load/write SKILL artifacts
• src/skills_evaluator/linting.py: lightweight skill lint checks
• src/skills_evaluator/placebo.py: sham skill generation
• src/skills_evaluator/harness.py: harness interfaces + heuristic/replay harnesses
• src/skills_evaluator/evaluator.py: matrix execution + aggregation
• src/skills_evaluator/metrics.py: derived effects and classification logic
• src/skills_evaluator/optimize.py: practical optimizer loop (local + optional GEPA)
• src/skills_evaluator/reporting.py: markdown report generation
• src/skills_evaluator/cli.py: CLI commands

Skill eval convention

Keep eval assets under the skill directory:

my-skill/
├── SKILL.md
├── references/
└── evals/
    ├── tasks.yaml
    └── ...

• evaluate and optimize default to evals/tasks.yaml under --skill-dir
• --tasks-file still works as an explicit override
• files under evals/ are ignored when loading skill support files, so they do not bloat skill context

Quick start

Use the included sample skill and tasks:

uv run skills-evaluator evaluate \
  --skill-dir examples/skills/sample-code-review \
  --report-file outputs/report.md \
  --json-file outputs/eval.json

Generate placebo control skill:

uv run skills-evaluator make-placebo \
  --skill-dir examples/skills/sample-code-review \
  --out-dir outputs/placebo-skill

Run local optimizer loop:

uv run skills-evaluator optimize \
  --skill-dir examples/skills/sample-code-review \
  --out-dir outputs/optimized-skill \
  --report-file outputs/optimized-report.md \
  --backend local

Optional GEPA backend

You can switch optimizer backend to gepa.

uv run skills-evaluator optimize \
  --skill-dir examples/skills/sample-code-review \
  --out-dir outputs/optimized-skill \
  --report-file outputs/optimized-report.md \
  --backend gepa \
  --gepa-reflection-model openai/gpt-5.1

Override the default dataset location when needed:

uv run skills-evaluator evaluate \
  --skill-dir skills/markit \
  --tasks-file outputs/custom-tasks.yaml \
  --report-file outputs/markit-report.md

If GEPA/model setup fails, the optimizer automatically falls back to local mutations.

CLI commands

• skills-evaluator lint-skill
• skills-evaluator make-placebo
• skills-evaluator evaluate
• skills-evaluator optimize

Research anchors (verified)

• Claude skill best practices (concise metadata, progressive disclosure, token trade-offs): https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices
• DSPy ReAct and GEPA APIs: https://dspy.ai/api/modules/ReAct/ https://dspy.ai/api/optimizers/GEPA/overview/
• GEPA optimize_anything evaluator + ASI pattern: https://gepa-ai.github.io/gepa/blog/2026/02/18/introducing-optimize-anything/ https://gepa-ai.github.io/gepa/api/optimize_anything/Evaluator/
• SkillsBench findings (curated skills can help; some tasks regress; focused skills often outperform broad docs): https://arxiv.org/abs/2602.12670
• Accuracy+cost and holdout discipline for agent evaluation: https://arxiv.org/abs/2407.01502