A real-time, local-first voice agent, built Jarvis-style.
Talk into your mic, get a spoken reply in under a second, and if you interrupt it mid-sentence it actually stops and listens to you instead of finishing its thought. Under the hood it's streaming speech-to-text, an LLM agent that can call real tools (check your GPU temp, set timers, save notes, launch apps), and streaming text-to-speech, all glued together over a WebSocket.
Most voice assistant demos are just record, wait a few seconds, reply. That's simple to build. Getting the following three things right is what actually made this project interesting:
- Barge-in. You can talk over the assistant and it stops immediately, not gracefully, not with a fade, immediately. The in-flight LLM call and TTS playback both get cancelled the instant you start speaking again.
- Everything streams. Speech-to-text streams as you talk. The LLM streams tokens as it thinks. TTS synthesizes and plays sentence by sentence, so audio starts while the model is still writing the rest of the reply.
- Three LLM backends behind one interface. AWS Bedrock, Ollama (fully local), or OpenAI. Switching is a one-line config change, no code touched.
Every latency number below was actually measured on the machine this was built on, not estimated.
Browser (React) FastAPI server (Python)
┌───────────────────┐ WS ┌──────────────────────────────────┐
│ AudioWorklet mic │──PCM16──▶│ VAD (Silero) → turn detection │
│ capture (20ms │ │ │ │
│ frames) │ │ ▼ │
│ │ │ STT (faster-whisper, CUDA) │
│ AudioContext │◀──PCM16──│ │ │
│ playback + hard- │ (per │ ▼ │
│ stop on barge-in │ sentence) │ LLMProvider.chat_with_tools() │
│ │ │ ├─ Bedrock (Claude/Nova) │
│ Live transcript + │◀─JSON────│ ├─ Ollama (local) │
│ latency HUD │ events │ └─ OpenAI │
└───────────────────┘ │ │ (tool calls -> app/tools)│
│ ▼ │
│ TTS (Piper, per-sentence, │
│ streamed as synthesized) │
└──────────────────────────────────┘
Every assistant turn runs inside one cancellable asyncio task. Mic audio keeps getting fed to the VAD continuously, even while the assistant is mid-reply, so the moment it detects new speech it cancels that task and tells the browser to cut audio immediately. That one decision shapes basically the whole backend (see app/pipeline.py).
| Layer | Choice | Why |
|---|---|---|
| STT | faster-whisper (distil-small.en, CUDA) | CTranslate2 engine, roughly 4x faster than plain Whisper |
| VAD | Silero VAD | Tiny and fast, the standard cheap way to detect speech vs silence |
| LLM | AWS Bedrock / Ollama / OpenAI, pluggable | See app/providers/ |
| TTS | Piper (ONNX, CPU) | Fast enough on CPU that it never has to fight STT/LLM for VRAM |
| Backend | FastAPI + WebSockets, Python 3.11, uv |
Async-native, one connection is one bidirectional stream |
| Frontend | React + TypeScript + Vite, hand-written AudioWorklet | Full control over mic timing instead of relying on a black-box library |
One line in .env, nothing else:
LLM_PROVIDER=bedrock # or: ollama | openaibedrock(default): uses AWS's unified Converse/ConverseStream API, works across Claude and Amazon Nova with the same code path, picks up your existing~/.aws/credentialsautomatically.ollama: fully local, zero API cost, works offline.openai: standard Chat Completions API, also works with any OpenAI-compatible endpoint (Groq, Together, a local llama.cpp server) viaOPENAI_BASE_URL.
All three implement the same small interface in app/providers/base.py, so pipeline.py never has to know which one it's actually talking to.
Measured on an RTX 5060 Ti 16GB / i7-14700K / Windows 11, see bench/ for the script.
| Stage | Time |
|---|---|
| STT (faster-whisper, distil-small.en, CUDA, warm) | 20 to 70ms per 10s of audio (RTF 0.02 to 0.07) |
LLM time to first token, Bedrock claude-3-haiku |
about 500ms |
LLM time to first token, Bedrock claude-sonnet-5 / opus-5 |
1.3 to 1.9s |
| TTS (Piper, CPU, warm) | about 60ms to synthesize a full sentence |
| VAD per-frame cost | about 0.2ms |
That's why the default model is claude-3-haiku and not one of the newer, smarter ones. For a voice agent, 3 to 4x more dead air before it starts talking is a bigger deal than a bit more reasoning depth. If you'd rather optimize for quality over speed, .env.example has the newer model IDs and how to switch.
get_system_stats (live CPU/RAM/GPU/VRAM off the machine it's running on), set_timer, save_note / recall_notes, launch_app (allowlisted), set_volume, web_search.
cd server
uv run pytest -v29 tests covering the VAD state machine, TTS synthesis, the provider tool-calling loop (tested against a scripted fake provider, no live API calls needed), the tool registry, and the WebSocket protocol schemas.
You'll need Python 3.11+, Node 18+, and ideally an NVIDIA GPU (it'll fall back to CPU for STT if you don't have one, just slower). For the LLM you need either AWS credentials configured (aws configure) if you're using the default Bedrock provider, or Ollama running locally if you switch to that.
1. Clone it
git clone https://github.com/aryeo0908/whisper-loop.git
cd whisper-loop2. Backend
cd server
uv sync
cp .env.example .envOpen .env and set things up for your setup (AWS region/model, or Ollama host, or OpenAI key, whichever provider you want). You'll also need the Piper voice model, drop en_US-lessac-medium.onnx and its .onnx.json into server/assets/voices/ (grab it from rhasspy/piper-voices on Hugging Face).
Then start the server:
uv run uvicorn app.main:app --reloadIt'll download the Whisper model on first run, that can take a minute depending on your connection.
3. Frontend
In a second terminal:
cd web
npm install
npm run devOpen http://localhost:5173, click "Start talking," and allow mic access when your browser asks.
4. Try the interrupt
Ask it something, and while it's mid-reply, just start talking again. It should stop instantly and start listening. That's the whole point of this project, so it's worth trying on purpose at least once.
MIT, see LICENSE.