Skip to content

test: live E2E smoke test; docs for alpha testers - #117

Merged
PaulAsjes merged 2 commits into
nextfrom
feat/e2e-smoke-and-docs
Jul 29, 2026
Merged

test: live E2E smoke test; docs for alpha testers#117
PaulAsjes merged 2 commits into
nextfrom
feat/e2e-smoke-and-docs

Conversation

@PaulAsjes

Copy link
Copy Markdown
Collaborator

Task 7, scoped to the E2E smoke test and docs. No version tag — nothing is released by this PR.

Live E2E smoke test

tests/e2e_smoke.rs runs the whole workflow against a real account: initaddpushpulldelete. Its core assertion is the one thing unit and wire tests can't reach: a config survives a push/pull round-trip byte-for-byte, which is the guarantee the raw-JSON design exists to provide. It also checks status reports the live id and that the widget snippet embeds it.

Safety — it creates and deletes agents, so:

  • Opt-in via ELEVENLABS_E2E_API_KEY, deliberately not ELEVENLABS_API_KEY. A shell with your normal credentials exported cannot mutate a live workspace by running cargo test. There's a test asserting that wiring stays that way.
  • Skips with a clear message when unset, so cargo test and CI are unaffected.
  • Redirects HOME at a temp dir, so it can't read or write your ~/.elevenlabs residency setting.
  • Always deletes the agent it created, including on panic; prints the id if deletion fails.
ELEVENLABS_E2E_API_KEY=xi-... cargo test --test e2e_smoke -- --nocapture

I deliberately did not wire it into CI — it's destructive, so it shouldn't fire automatically on every push.

A bug the pre-flight caught

Before shipping I checked each invocation's arg shape against the real binary, and the cleanup step was wrong: generated API commands take path parameters as flags, so deletion is agents delete --agent-id <id>, not a positional. As written the cleanup would have failed and leaked agents in the test account on every run.

Docs

  • "Two surfaces in one namespace"elevenlabs agents holds both workflow verbs (positional args, operate on local files, progress output) and generated API commands (flag args, one request each, --format-able). Every claim in that table was verified against the built binary.
  • v0 migration notes that fall out of it: agents list is now the API's list (use agents status for the local view), agents delete --agent-id <id> (and it no longer cleans up locally), and agents widget <id>agents widget embed <id>.
  • Installation reworked to lead with building from source, with a pre-release banner — the existing curl | sh instructions pointed at releases/latest, which doesn't exist yet and would have failed for alpha testers.
  • Development section covering the three test suites (including why the generated crates need separate --manifest-path runs) and the E2E warning.
  • Dropped --locked from the install instructions after testing it: the committed Cargo.lock is incomplete, so cargo install --locked errors out.

tests/e2e_smoke.rs is added to .fernignore (its neighbour wire_test.rs is generated and stays regenerable).

Verified

Full suite green: 1752 framework + 47 workflow + 345 wire + 2 e2e (skipped without a key) + doc tests. The E2E test's own compile and skip path confirmed; every documented command checked against the binary.

Note

Cargo.lock is left out of this PR — it has ~30 packages of pre-existing churn that appear on any build, unrelated to these changes. Worth a separate lockfile refresh (and it's what makes --locked fail).

🤖 Generated with Claude Code

PaulAsjes and others added 2 commits July 28, 2026 19:31
E2E (tests/e2e_smoke.rs): runs the whole workflow against a real account —
init → add → push → pull → delete — and asserts a config survives a push/pull
round-trip byte-for-byte, which is the guarantee the raw-JSON design in
workflow/api.rs exists to provide and the one thing unit and wire tests can't
check.

Opt-in via ELEVENLABS_E2E_API_KEY, deliberately *not* ELEVENLABS_API_KEY: the
test creates and deletes agents, so a shell with normal credentials exported
must not be able to mutate a live workspace by running `cargo test`. It skips
with a message when unset, redirects HOME at a temp dir so it can't read or
write the developer's residency config, and always deletes the agent it
created (printing the id if that fails).

Pre-flighting the invocations caught a bug in the cleanup step: generated API
commands take path parameters as flags, so deletion is
`agents delete --agent-id <id>`, not a positional. As written it would have
failed and leaked agents in the test account.

Docs: added a "Two surfaces in one namespace" section — the agents group holds
both workflow verbs (positional args, operate on local files) and generated API
commands (flag args, one request each) — plus the v0 migration notes that fall
out of it (`agents list` → `agents status` for the local view, `delete
--agent-id`, `widget` → `widget embed`). Reworked Installation to lead with
building from source, since there are no published binaries yet, and added a
Development section covering the three test suites and the E2E safety warning.
Dropped `--locked` from the install instructions: the committed Cargo.lock is
incomplete, so it fails.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the pre-release framing: installers lead again, "Build from source" is a
normal third option rather than the alpha-tester path, and nothing hedges about
a future first release.

Rewrite Authentication, which was the generated placeholder ("Run
`elevenlabs --help` for details"). Checking what the CLI actually does turned up
that the framework's `auth login` / `logout` / `status` are inert here: the spec
declares no securitySchemes, only an `x-fern-global-headers` entry, so
`auth status` reports "No auth schemes are declared on this CLI" and
`auth login --with-token` fails outright. Authentication is the `xi-api-key`
header, sourced from ELEVENLABS_API_KEY (a .env file is picked up too) or the
per-command --xi-api-key flag. All three verified against --dry-run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@PaulAsjes
PaulAsjes merged commit 360cd1a into next Jul 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant