Skip to content

feat: tests command group (add/delete/push/pull + auto-discovery) - #113

Merged
PaulAsjes merged 2 commits into
nextfrom
feat/tests-group
Jul 28, 2026
Merged

feat: tests command group (add/delete/push/pull + auto-discovery)#113
PaulAsjes merged 2 commits into
nextfrom
feat/tests-group

Conversation

@PaulAsjes

Copy link
Copy Markdown
Collaborator

What

Ports v0's src/tests/ as the tests command group — the last of the three entity groups, completing v0 parity for agents/tools/tests.

  • tests add <name> [--template …] — build from a template, create in ElevenLabs, write test_configs/<name>.json, register in tests.json. All four v0 templates ported: basic-llm, tool, conversation-flow, customer-service.
  • tests push [--test <id>] [--config-dir …] [--dry-run] — force-override create/update.
  • tests pull [--test <id>] [--output-dir] [--update] [--all] [--dry-run] — create/update/skip planning, same shape as agents/tools.
  • tests delete <id> / --all
  • tests templates list

The interesting bit: push-time auto-discovery

tests push scans --config-dir recursively for .json files that aren't yet in tests.json and structurally look like tests (a chat_history array or a success_condition string), registers them, and warns on non-test JSON. So you can drop a config into test_configs/ and push without hand-editing the index. Ports v0's registerDiscoveredTestConfigs/isTestConfigFile, including recursive walk and posix-normalized paths for portable index entries.

Verified end-to-end offline:

$ tests push --dry-run     # after dropping 3 files into test_configs/
Warning: Skipping non-test JSON file test_configs/not-a-test.json
Discovered 2 test config(s) in test_configs      # incl. a nested subdir
Greeting: Will push (force override)
[DRY RUN] Would create test: Greeting
Deep: Will push (force override)
[DRY RUN] Would create test: Deep

…and re-running with one already tracked discovers only the other, correctly showing update for the tracked one and create for the new one.

Also

  • api.rs: tests API against v1/convai/agent-testing via the raw-JSON layer (lossless round-trip); list paginates to completion.
  • project.rs: discover_json_files() helper (recursive, sorted, posix-normalized).

Verified

cargo build green; 19 workflow unit tests pass (5 new: all templates build, unknown-template error, tool template shape, multi-turn flow, and the test-config shape classifier). Offline smoke tests cover discovery, the --test filter, and template validation. (Live create/update/pull needs an API key.)

Remaining for v1

residency + components add, then release readiness (port remaining v0 unit tests, E2E smoke, docs, tag 1.0.0).

🤖 Generated with Claude Code

PaulAsjes and others added 2 commits July 28, 2026 12:17
Port v0's src/tests/ as the `tests` command group, completing v0 feature
parity for the entity groups.

- api.rs: create_test/update_test/get_test/list_tests/delete_test against
  v1/convai/agent-testing, via the raw-JSON layer (configs round-trip
  losslessly). list paginates to completion.
- project.rs: discover_json_files() walks a config dir recursively and
  normalizes paths to posix separators so index entries stay portable.
- tests.rs: add (4 templates ported from v0: basic-llm, tool,
  conversation-flow, customer-service), delete (single + --all), push, pull
  (create/update/skip planning), and `tests templates list`.
- push-time auto-discovery: scans --config-dir for untracked .json files
  that structurally look like tests (chat_history array or success_condition
  string), registers them in tests.json, and warns on non-test JSON — so a
  dropped-in config pushes without hand-editing the index.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@PaulAsjes
PaulAsjes merged commit 60bb411 into next Jul 28, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant