Skip to content

Commit e01dbf4

Browse files
authored
Refresh media providers and repair Google and ElevenLabs transports (OpenMontageInc#845)
1 parent 0f191f9 commit e01dbf4

46 files changed

Lines changed: 6221 additions & 525 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/elevenlabs/SKILL.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,13 @@ description: Generate AI voiceovers, sound effects, and music using ElevenLabs A
55

66
# ElevenLabs Audio Generation
77

8+
## Current v4 transports
9+
10+
For v4, v4 Turbo, Instant Soundtracks and realtime models, read
11+
[the October integration guide](../provider-refresh-oct23/SKILL.md).
12+
v4 uses Text to Dialogue; Turbo uses the dialogue WebSocket.
13+
Do not send those models to the legacy text-to-speech endpoint.
14+
815
## OpenMontage provider routing
916

1017
Inspect the OpenMontage registry before choosing an authentication path.

‎.agents/skills/provider-model-refresh/SKILL.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,3 +103,10 @@ Contracts were inspected 2026-10-03; each schema records its official source.
103103
See docs/provider-update-plan-2026-10-03.md for release research. Public Kling 4
104104
generation endpoints remain unverified, so do not invent a model ID. Live voice
105105
agents require a separate streaming/session integration beyond batch narration.
106+
107+
## October 23 follow-up
108+
109+
For current Google retirement handling, Eleven v4 transports, and newly supported
110+
providers, read [provider-refresh-oct23](../provider-refresh-oct23/SKILL.md).
111+
The v4 narration HTTP payload is dialogue inputs/settings, not legacy TTS
112+
text/voice_settings; v4 Turbo is WebSocket-only in the direct file adapter.
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
name: provider-refresh-oct23
3+
description: API contracts and production routing for the October 23, 2026 provider refresh.
4+
---
5+
6+
# October 23 provider integrations
7+
8+
Read the tool's current input schema before use. Announce tool, host, exact model,
9+
operation and quoted cost before production. No automatic host/model fallback.
10+
Use explicit output paths under the production project. Account access is not
11+
established merely by having a credential configured. Most new prices require a
12+
fresh account quote; do not use expired promotions as permanent prices.
13+
14+
## Google and ElevenLabs migrations
15+
16+
- `gemini_omni_video`: direct Gemini Interactions API model is
17+
`gemini-omni-1.1-flash`. Old preview retired October 22. Continue editing with
18+
`previous_interaction_id`; do not substitute it inside a Veo request.
19+
- `veo_video`: Google backend requires Vertex; fal retains its own contracts.
20+
Gemini API Veo has retired. Select Omni explicitly for the replacement workflow.
21+
- `google_imagen`: new default `gemini-nano-banana-2.1`; Imagen models require
22+
Vertex. Native Gemini image models use generate_content, not Imagen predict.
23+
- `elevenlabs_tts`: `eleven_v4` uses HTTP Text to Dialogue with `inputs` and
24+
`settings`. Keep each narration request <=2,000 characters. `next_text` maps to
25+
`future_text`; context strings <=100 characters. Style/speed/speaker boost are
26+
not dialogue settings. `eleven_v4_turbo` uses the dialogue WebSocket; this file
27+
adapter does not support timestamps or continuity controls on Turbo. It waits
28+
for global `is_final`, not the per-turn marker. The current official realtime
29+
tutorial documents Turbo although the older WebSocket reference says v3-only.
30+
31+
## Narration and conversion
32+
33+
- `heygen_tts`: model_id `heygen-voice-1`, `heygen-voice-1-turbo`, or
34+
`heygen-voice-1-turbo-free`. Pass an ACTIVE native public/workspace voice ID;
35+
`operation=list_voices` discovers native voices. Professional clones only work
36+
with the non-Turbo model. `voice_type` can prevalidate the category; the service
37+
validates actual ownership/status. Completed speech and SSE both write WAV.
38+
SSE parts are individual WAV containers, decoded before assembly. Timestamps
39+
become an alignment sidecar and word_timestamps. Free access ends November 30,
40+
2026; failures never switch to paid. Rate rejection retries are bounded and honor
41+
short Retry-After delays; larger delays surface to the caller.
42+
- `heygen_catalog_tts`: separate catalog IDs, `/v3/voices/speech`; explicitly choose
43+
`engine` (starfish, heygen_voice, elevenlabs). Native professional clone IDs do
44+
not belong here. Catalog output can be MP3 or WAV. Supply either suffix; the returned artifact
45+
path uses the actual response codec suffix and preserves the audio.
46+
- `heygen_voice_convert`: ACTIVE instant workspace voice only, at most300 seconds.
47+
Exactly one audio_path (<=16 MB), audio_url or asset_id. Output WAV. Supply a
48+
stable idempotency_key if resuming after an ambiguous timeout; no automatic retry.
49+
- `grok_tts`: xAI narration `/v1/tts`, not Grok Voice realtime. `language=auto` by
50+
default, voice_id defaults to eve; list_voices discovers voices. output_format
51+
is an object, e.g. {"codec":"wav","sample_rate":24000}; this file adapter exposes
52+
MP3/WAV, not headerless telephony codecs. Optional character alignment is saved.
53+
- `mistral_tts`: model_id `voxtral-mini-tts-3-0`; exactly one voice_id or local
54+
reference_audio_path. mp3/wav/flac/opus. Inline emotion/pace tags and expressiveness;
55+
free-form style instructions are not available at launch. No realtime TTS claim.
56+
- `mai_tts`: Azure Speech key/region, model_id MAI-Voice-2.1 or MAI-Voice-2.1-Flash.
57+
voice_id is a managed voice base such as en-US-Harper (the model suffix is added).
58+
The TTS selector translates speed and pitch through the Azure SSML adapter.
59+
This adapter does not provision personal voice clones or grant gated access.
60+
- `mistral_stt`: Voxtral Mini/Small Transcribe3, local audio files; timestamps are
61+
segments, not words. Diarization implies segment timestamps. Raw transcript JSON
62+
is persisted, including speaker and usage metadata.
63+
- `elevenlabs_soundtrack`: video_paths (<=10, combined <=200 MB / 600seconds),
64+
optional description/tags, add_sfx/use_audio_as_reference/separate_tracks. Separate
65+
tracks return playable files in a sibling *_tracks directory; never treat a ZIP
66+
as an audio file. The adapter rejects unsafe archive paths and oversized archives.
67+
68+
## Images and video
69+
70+
| Model | Explicit tools / hosts |
71+
|---|---|
72+
| Nano Banana2.1 | google_imagen, gemini_fal_image, gemini_replicate_image |
73+
| FLUX3 Image | bfl_image, flux3_fal_image, flux3_replicate_image |
74+
| FLUX3 Max video | flux3_video (fal) |
75+
| Vidu Q4 | vidu_video (fal), image_to_video or reference_to_video; no text-only route |
76+
| Fibo2 | bria_image direct or bria_fal_image |
77+
| MAI Image2.6 / Flash | mai_image, deployed Foundry models |
78+
| Grok Imagine1.5 / Lite | grok_video direct xAI |
79+
80+
fal and Replicate have checked-in endpoint schemas under tools/provider_contracts.
81+
Pass provider_params only when documented by that host. Do not copy fal fields to
82+
Replicate or direct APIs. Existing Pro image model defaults remain unchanged on
83+
aggregators; select the new model explicitly.
84+
85+
BFL bounding-box layouts are encoded in the prompt per BFL docs, not a fabricated
86+
`boxes` endpoint field. BFL supports up to10 reference images; width/height must be
87+
provided together. Its API reference is authoritative over older narrative examples.
88+
Bria persists structured metadata for later editing. Its shared endpoint can route
89+
some legacy combinations to older Fibo models; the direct tool rejects those
90+
combinations when advertising Fibo2. MAI requires model identity to match the
91+
configured deployment; PNG output and <=2,359,296 pixels with each side >=768.
92+
93+
Grok1.5 references are limited to720p. Lite has no reference audio, reference images,
94+
last_frame or keyframes. Full1.5 supports preset voice references; custom audio
95+
references remain partner gated. Mid-video pins must be inside duration and occupy
96+
different1/3-second slots. No model substitution on failure. fal/Replicate/BFL/Grok
97+
jobs expose resume metadata; save job_path before polling and resume pending jobs
98+
instead of submitting duplicate paid generations.
99+
100+
## Realtime is separate from narration
101+
102+
`realtime_audio` is a bounded one-response file runner for Grok Voice3,
103+
Eleven Realtime Mini/Max and MAI-Transcribe-2-Streaming. Input is text or mono PCM16
104+
WAV (16/24kHz; Eleven requires24kHz). Voice output is WAV; MAI output is event JSON.
105+
Input sending and event receiving run concurrently. MAI commits explicitly and
106+
waits for final transcription, not a commit acknowledgment. Eleven uses ephemeral
107+
session credentials and bursty input pacing for recorded files.
108+
109+
For live audio, VAD, interruptions, multiple turns, or function calls, use the
110+
context-managed `tools.audio._realtime_session.RealtimeAudioSession` transport:
111+
`append_audio`, `commit`, `receive`, `request_response`, `cancel`, and
112+
`submit_tool_result`. The caller owns the session lifecycle and tool approvals.
113+
Received tool calls are never executed automatically. The finite file runner
114+
rejects tools because it cannot continue them after closing the socket. No microphone
115+
UI, browser app, or voice-agent orchestration is implied by this transport layer.
116+
Secrets remain in memory. No realtime translation or Kling4 support was added.
117+
118+
## Primary references
119+
120+
- https://ai.google.dev/gemini-api/docs/deprecations
121+
- https://elevenlabs.io/docs/api-reference/text-to-dialogue/convert
122+
- https://elevenlabs.io/docs/eleven-api/guides/how-to/websockets/realtime-tdd
123+
- https://elevenlabs.io/docs/api-reference/music/video-to-music
124+
- https://developers.heygen.com/docs/voices/heygen-voice-speech
125+
- https://developers.heygen.com/docs/voices/speech-to-speech
126+
- https://developers.heygen.com/reference/generate-speech
127+
- https://docs.x.ai/developers/model-capabilities/audio/text-to-speech
128+
- https://docs.x.ai/developers/model-capabilities/video/overview
129+
- https://docs.mistral.ai/studio/audio/overview
130+
- https://docs.bfl.ml/api-reference/utility/generate-an-image-with-flux-3
131+
- https://docs.bria.ai/whats-new/fibo-2
132+
- https://learn.microsoft.com/azure/ai-services/speech-service/mai-voices
133+
- https://learn.microsoft.com/azure/foundry/foundry-models/how-to/use-foundry-models-mai-image
134+
- https://learn.microsoft.com/azure/ai-services/speech-service/mai-transcribe-2-streaming-realtime
135+
- https://elevenlabs.io/docs/eleven-agents/guides/openai-realtime-compatibility
136+
- https://docs.x.ai/developers/rest-api-reference/inference/voice

0 commit comments

Comments
 (0)