test: live E2E smoke test; docs for alpha testers - #117
Merged
Conversation
E2E (tests/e2e_smoke.rs): runs the whole workflow against a real account — init → add → push → pull → delete — and asserts a config survives a push/pull round-trip byte-for-byte, which is the guarantee the raw-JSON design in workflow/api.rs exists to provide and the one thing unit and wire tests can't check. Opt-in via ELEVENLABS_E2E_API_KEY, deliberately *not* ELEVENLABS_API_KEY: the test creates and deletes agents, so a shell with normal credentials exported must not be able to mutate a live workspace by running `cargo test`. It skips with a message when unset, redirects HOME at a temp dir so it can't read or write the developer's residency config, and always deletes the agent it created (printing the id if that fails). Pre-flighting the invocations caught a bug in the cleanup step: generated API commands take path parameters as flags, so deletion is `agents delete --agent-id <id>`, not a positional. As written it would have failed and leaked agents in the test account. Docs: added a "Two surfaces in one namespace" section — the agents group holds both workflow verbs (positional args, operate on local files) and generated API commands (flag args, one request each) — plus the v0 migration notes that fall out of it (`agents list` → `agents status` for the local view, `delete --agent-id`, `widget` → `widget embed`). Reworked Installation to lead with building from source, since there are no published binaries yet, and added a Development section covering the three test suites and the E2E safety warning. Dropped `--locked` from the install instructions: the committed Cargo.lock is incomplete, so it fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the pre-release framing: installers lead again, "Build from source" is a
normal third option rather than the alpha-tester path, and nothing hedges about
a future first release.
Rewrite Authentication, which was the generated placeholder ("Run
`elevenlabs --help` for details"). Checking what the CLI actually does turned up
that the framework's `auth login` / `logout` / `status` are inert here: the spec
declares no securitySchemes, only an `x-fern-global-headers` entry, so
`auth status` reports "No auth schemes are declared on this CLI" and
`auth login --with-token` fails outright. Authentication is the `xi-api-key`
header, sourced from ELEVENLABS_API_KEY (a .env file is picked up too) or the
per-command --xi-api-key flag. All three verified against --dry-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 7, scoped to the E2E smoke test and docs. No version tag — nothing is released by this PR.
Live E2E smoke test
tests/e2e_smoke.rsruns the whole workflow against a real account:init→add→push→pull→delete. Its core assertion is the one thing unit and wire tests can't reach: a config survives a push/pull round-trip byte-for-byte, which is the guarantee the raw-JSON design exists to provide. It also checksstatusreports the live id and that the widget snippet embeds it.Safety — it creates and deletes agents, so:
ELEVENLABS_E2E_API_KEY, deliberately notELEVENLABS_API_KEY. A shell with your normal credentials exported cannot mutate a live workspace by runningcargo test. There's a test asserting that wiring stays that way.cargo testand CI are unaffected.HOMEat a temp dir, so it can't read or write your~/.elevenlabsresidency setting.ELEVENLABS_E2E_API_KEY=xi-... cargo test --test e2e_smoke -- --nocaptureI deliberately did not wire it into CI — it's destructive, so it shouldn't fire automatically on every push.
A bug the pre-flight caught
Before shipping I checked each invocation's arg shape against the real binary, and the cleanup step was wrong: generated API commands take path parameters as flags, so deletion is
agents delete --agent-id <id>, not a positional. As written the cleanup would have failed and leaked agents in the test account on every run.Docs
elevenlabs agentsholds both workflow verbs (positional args, operate on local files, progress output) and generated API commands (flag args, one request each,--format-able). Every claim in that table was verified against the built binary.agents listis now the API's list (useagents statusfor the local view),agents delete --agent-id <id>(and it no longer cleans up locally), andagents widget <id>→agents widget embed <id>.curl | shinstructions pointed atreleases/latest, which doesn't exist yet and would have failed for alpha testers.--manifest-pathruns) and the E2E warning.--lockedfrom the install instructions after testing it: the committedCargo.lockis incomplete, socargo install --lockederrors out.tests/e2e_smoke.rsis added to.fernignore(its neighbourwire_test.rsis generated and stays regenerable).Verified
Full suite green: 1752 framework + 47 workflow + 345 wire + 2 e2e (skipped without a key) + doc tests. The E2E test's own compile and skip path confirmed; every documented command checked against the binary.
Note
Cargo.lockis left out of this PR — it has ~30 packages of pre-existing churn that appear on any build, unrelated to these changes. Worth a separate lockfile refresh (and it's what makes--lockedfail).🤖 Generated with Claude Code