A single native C++ server that exposes RapidSpeech.cpp's ASR and TTS pipelines over three protocols:
- OpenAI-compatible HTTP API — drop-in for OpenAI audio clients:
POST /v1/audio/transcriptions— audio (WAV) → text (ASR)POST /v1/audio/speech— text → audio (WAV) (TTS)GET /v1/models,GET /health
- Model Context Protocol (MCP), JSON-RPC 2.0 — tools for LLM agents:
transcribe(audio file path → text),synthesize(text → WAV file)- transports: stdio (Claude Desktop et al.) and HTTP (
POST /mcp)
- WebSocket streaming — four low-latency endpoints (see below):
/asr/stream,/asr/vad,/tts/segmented,/tts/stream
One process serves ASR, TTS, or both. Each model context is mutex-guarded, so concurrent HTTP requests are serialized per model.
Dependencies are vendored single headers: httplib.h
(cpp-httplib, MIT) for HTTP, rs_ws.h (self-contained RFC 6455
WebSocket) for streaming, and rs_json.h (self-contained) for
JSON. No Python, no external services.
cmake -B build && cmake --build build --target rs-server -j
# → build/rs-server# Both ASR and TTS
./build/rs-server \
--asr-model xasr-q4_k_m.gguf \
--tts-model omnivoice-f16.gguf \
--vad-model silero_vad_v6.gguf \
--host 127.0.0.1 --port 8080 # WebSocket on 8081
# ASR only / TTS only — just omit the other flag
./build/rs-server --asr-model xasr-q4_k_m.gguf --port 8080Options: --asr-model, --tts-model, --vad-model (for /asr/vad),
--vad-threshold (default 0.5), --vad-partial-ms (default 600, 0=off),
--vad-end-silence-ms (default 600; larger values merge short pauses),
--vad-min-segment-ms (default 500), --vad-max-segment-ms (default 0=off),
--two-pass (FunASR-Nano CTC first pass + LLM final re-decode),
--ctc-precheck (skip LLM on CTC-empty silence/noise),
--mem-log (log ASR scheduler compute-buffer high-water marks),
--asr-reload-every N (reload ASR after N final segments to release backend
buffers; default 0=off),
--host (default 127.0.0.1), --port (8080), --ws-port (default port+1),
--web-dir (serve a static UI dir at /), --mcp-stdio, -t/--threads (4),
--cpu, -h.
webui.html is a self-contained browser UI (no build step) that
exercises every endpoint: file/mic transcription, streaming ASR with live
partials, VAD segments, and streaming TTS playback. Serve it from the server so
it's same-origin (needed for microphone access):
./build/rs-server --asr-model xasr.gguf --tts-model tts.gguf \
--vad-model silero.gguf --web-dir examples/server --port 8080
# open http://127.0.0.1:8080/webui.html(CORS is enabled, so you can also just open the file directly — HTTP calls work,
but microphone capture requires the same-origin --web-dir route.)
curl http://127.0.0.1:8080/v1/audio/transcriptions \
-F file=@audio.wav \
-F model=rapidspeech-asr
# → {"text":"..."}
# add -F response_format=text for plain textFor long WAV files, load a VAD model and ask the HTTP endpoint to split speech before ASR:
curl "http://127.0.0.1:8080/v1/audio/transcriptions" \
-F file=@long.wav \
-F model=rapidspeech-asr \
-F vad=true \
-F vad_end_silence_ms=900 \
-F vad_min_segment_ms=300 \
-F vad_max_segment_ms=15000
# → {"text":"...","vad":true,"segments":[{"id":0,"start":...,"end":...,"text":"..."}]}HTTP VAD mode uses the same server-wide VAD defaults as /asr/vad, and accepts
the same override fields: vad_threshold, vad_end_silence_ms,
vad_min_segment_ms, vad_max_segment_ms, two_pass, and ctc_precheck.
The fields can be sent either as multipart form fields or URL query parameters.
With the official OpenAI Python client:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
with open("audio.wav", "rb") as f:
print(client.audio.transcriptions.create(model="rapidspeech-asr", file=f).text)curl http://127.0.0.1:8080/v1/audio/speech \
-H "content-type: application/json" \
-d '{"model":"rapidspeech-tts","input":"你好,欢迎使用 RapidSpeech","voice":"female","language":"Chinese"}' \
--output out.wavRecognized body fields: input (required), voice (→ instruct/speaker),
language, seed, n_steps (TTS diffusion steps), plus instruct as an
explicit alias for voice. Output is a 16-bit mono WAV at the model's rate.
./build/rs-server --asr-model xasr.gguf --tts-model tts.gguf --mcp-stdioIn stdio mode the JSON-RPC stream owns stdout; all framework/ggml logs are diverted to stderr so they never corrupt it.
Claude Desktop config (claude_desktop_config.json):
{
"mcpServers": {
"rapidspeech": {
"command": "/abs/path/to/build/rs-server",
"args": ["--asr-model", "/abs/xasr.gguf",
"--tts-model", "/abs/tts.gguf",
"--mcp-stdio"]
}
}
}Then ask the agent to "transcribe /path/to/audio.wav" or "synthesize 'hello' to /tmp/hi.wav".
The same JSON-RPC is available at POST /mcp (one message per request):
curl -X POST http://127.0.0.1:8080/mcp \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -X POST http://127.0.0.1:8080/mcp \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/call",
"params":{"name":"transcribe","arguments":{"path":"/abs/audio.wav"}}}'Tools:
| Tool | Arguments | Result |
|---|---|---|
transcribe |
path (WAV file) |
transcribed text |
synthesize |
text, optional output_path / voice / language |
writes a WAV, returns its path + duration |
WebSocket endpoints run on a separate port (--ws-port, default HTTP
port + 1). Wire format: binary frames = raw 16-bit signed little-endian mono
PCM; text frames = JSON (control input, and result/marker output). The
server serializes one session per model (ggml contexts are single-threaded); a
second concurrent session on the same model is rejected with
{"type":"error","error":"...busy"}.
| Endpoint | Direction | Purpose |
|---|---|---|
/asr/stream |
PCM in → text out | Pure streaming ASR (X-ASR). partial per token update, final on {"type":"eos"}/close. Optional ?chunk_len=16|32|48|96. |
/asr/vad |
PCM in → text out | VAD-segmented offline ASR. partial while a segment grows, final per segment. Needs --vad-model. |
/tts/segmented |
text in → audio out | Splits input into sentences, synthesizes each fully, streams segment marker + audio per sentence. Works for any TTS model. |
/tts/stream |
text in → audio out | Pure streaming TTS — emits each audio chunk as generated (low first-chunk latency on CosyVoice3). |
/asr/vad uses the server-wide VAD settings from the command line. A client can
override them for one WebSocket connection with query parameters:
ws://127.0.0.1:8081/asr/vad?vad_threshold=0.45&vad_end_silence_ms=900&vad_partial_ms=1000&vad_max_segment_ms=10000
Supported query keys: vad_threshold, vad_end_silence_ms, vad_partial_ms,
vad_min_segment_ms, vad_max_segment_ms, two_pass, and ctc_precheck.
Short aliases without the vad_ prefix are also accepted for VAD timings.
two_pass makes partials use the fast CTC pass and finals use LLM re-decode.
Message shapes:
Minimal Python client (streaming ASR):
import asyncio, json, wave, websockets
async def main():
wf = wave.open("audio_16k.wav", "rb") # 16 kHz mono s16le
async with websockets.connect("ws://127.0.0.1:8081/asr/stream") as ws:
assert json.loads(await ws.recv())["type"] == "ready"
async def reader():
async for m in ws:
print(json.loads(m)) # partial / final
task = asyncio.create_task(reader())
while (frames := wf.readframes(3200)): # 200 ms chunks
await ws.send(frames) # binary PCM
await asyncio.sleep(0.2) # pace ~ real time
await ws.send(json.dumps({"type": "eos"}))
await asyncio.sleep(1); task.cancel()
asyncio.run(main())Streaming TTS: send one text frame, then read a binary audio frame per chunk
(concatenate + wrap in a WAV header at the sample_rate from audio_info).
- Audio input is WAV (any rate/bit-depth; auto-resampled to the model's
rate). mp3/other formats aren't decoded — transcode to WAV first, or extend
decode_audio()inrs-server.cppto use miniaudio. - Concurrency: HTTP is multi-threaded, but each model runs one request at a time (ggml contexts are single-threaded); requests queue on a per-model mutex. WebSocket sessions hold the model for the whole session — a second concurrent session on the same model is rejected as "busy" (one live stream per model).
/asr/streamneeds an X-ASR model (true chunked streaming); other ASR models reply with an error on that endpoint (use/asr/vador the HTTP API instead)./tts/streamstreams per-chunk only on CosyVoice3; other TTS models emit a single chunk at the end (use/tts/segmentedfor sentence-level streaming).- GPU memory: ggml/CUDA keeps model weights and scheduler compute buffers
resident for reuse, so
nvidia-smiwill not drop after each segment. For long VAD runs, setvad_max_segment_msto bound the largest offline ASR graph. Use--mem-logto check whether internal compute buffers plateau. If you need the process to return CUDA buffers during a soak test, use--asr-reload-every N; this trades a reload pause for lower high-water memory. - This is an example server (plain HTTP, no auth/TLS/rate-limiting). Put it behind a reverse proxy for anything exposed.
- HTTP ASR uses the offline whole-clip path by default. For long audio, pass
vad=truewith--vad-modelto segment before ASR, or use/asr/streamwith an X-ASR model for true streaming.