feat: tests command group (add/delete/push/pull + auto-discovery) - #113
Merged
Conversation
Port v0's src/tests/ as the `tests` command group, completing v0 feature parity for the entity groups. - api.rs: create_test/update_test/get_test/list_tests/delete_test against v1/convai/agent-testing, via the raw-JSON layer (configs round-trip losslessly). list paginates to completion. - project.rs: discover_json_files() walks a config dir recursively and normalizes paths to posix separators so index entries stay portable. - tests.rs: add (4 templates ported from v0: basic-llm, tool, conversation-flow, customer-service), delete (single + --all), push, pull (create/update/skip planning), and `tests templates list`. - push-time auto-discovery: scans --config-dir for untracked .json files that structurally look like tests (chat_history array or success_condition string), registers them in tests.json, and warns on non-test JSON — so a dropped-in config pushes without hand-editing the index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports v0's
src/tests/as thetestscommand group — the last of the three entity groups, completing v0 parity for agents/tools/tests.tests add <name> [--template …]— build from a template, create in ElevenLabs, writetest_configs/<name>.json, register intests.json. All four v0 templates ported:basic-llm,tool,conversation-flow,customer-service.tests push [--test <id>] [--config-dir …] [--dry-run]— force-override create/update.tests pull [--test <id>] [--output-dir] [--update] [--all] [--dry-run]— create/update/skip planning, same shape as agents/tools.tests delete <id>/--alltests templates listThe interesting bit: push-time auto-discovery
tests pushscans--config-dirrecursively for.jsonfiles that aren't yet intests.jsonand structurally look like tests (achat_historyarray or asuccess_conditionstring), registers them, and warns on non-test JSON. So you can drop a config intotest_configs/and push without hand-editing the index. Ports v0'sregisterDiscoveredTestConfigs/isTestConfigFile, including recursive walk and posix-normalized paths for portable index entries.Verified end-to-end offline:
…and re-running with one already tracked discovers only the other, correctly showing
updatefor the tracked one andcreatefor the new one.Also
api.rs: tests API againstv1/convai/agent-testingvia the raw-JSON layer (lossless round-trip);listpaginates to completion.project.rs:discover_json_files()helper (recursive, sorted, posix-normalized).Verified
cargo buildgreen; 19 workflow unit tests pass (5 new: all templates build, unknown-template error, tool template shape, multi-turn flow, and the test-config shape classifier). Offline smoke tests cover discovery, the--testfilter, and template validation. (Live create/update/pull needs an API key.)Remaining for v1
residency+components add, then release readiness (port remaining v0 unit tests, E2E smoke, docs, tag1.0.0).🤖 Generated with Claude Code