|
| 1 | +--- |
| 2 | +name: provider-refresh-oct23 |
| 3 | +description: API contracts and production routing for the October 23, 2026 provider refresh. |
| 4 | +--- |
| 5 | + |
| 6 | +# October 23 provider integrations |
| 7 | + |
| 8 | +Read the tool's current input schema before use. Announce tool, host, exact model, |
| 9 | +operation and quoted cost before production. No automatic host/model fallback. |
| 10 | +Use explicit output paths under the production project. Account access is not |
| 11 | +established merely by having a credential configured. Most new prices require a |
| 12 | +fresh account quote; do not use expired promotions as permanent prices. |
| 13 | + |
| 14 | +## Google and ElevenLabs migrations |
| 15 | + |
| 16 | +- `gemini_omni_video`: direct Gemini Interactions API model is |
| 17 | + `gemini-omni-1.1-flash`. Old preview retired October 22. Continue editing with |
| 18 | + `previous_interaction_id`; do not substitute it inside a Veo request. |
| 19 | +- `veo_video`: Google backend requires Vertex; fal retains its own contracts. |
| 20 | + Gemini API Veo has retired. Select Omni explicitly for the replacement workflow. |
| 21 | +- `google_imagen`: new default `gemini-nano-banana-2.1`; Imagen models require |
| 22 | + Vertex. Native Gemini image models use generate_content, not Imagen predict. |
| 23 | +- `elevenlabs_tts`: `eleven_v4` uses HTTP Text to Dialogue with `inputs` and |
| 24 | + `settings`. Keep each narration request <=2,000 characters. `next_text` maps to |
| 25 | + `future_text`; context strings <=100 characters. Style/speed/speaker boost are |
| 26 | + not dialogue settings. `eleven_v4_turbo` uses the dialogue WebSocket; this file |
| 27 | + adapter does not support timestamps or continuity controls on Turbo. It waits |
| 28 | + for global `is_final`, not the per-turn marker. The current official realtime |
| 29 | + tutorial documents Turbo although the older WebSocket reference says v3-only. |
| 30 | + |
| 31 | +## Narration and conversion |
| 32 | + |
| 33 | +- `heygen_tts`: model_id `heygen-voice-1`, `heygen-voice-1-turbo`, or |
| 34 | + `heygen-voice-1-turbo-free`. Pass an ACTIVE native public/workspace voice ID; |
| 35 | + `operation=list_voices` discovers native voices. Professional clones only work |
| 36 | + with the non-Turbo model. `voice_type` can prevalidate the category; the service |
| 37 | + validates actual ownership/status. Completed speech and SSE both write WAV. |
| 38 | + SSE parts are individual WAV containers, decoded before assembly. Timestamps |
| 39 | + become an alignment sidecar and word_timestamps. Free access ends November 30, |
| 40 | + 2026; failures never switch to paid. Rate rejection retries are bounded and honor |
| 41 | + short Retry-After delays; larger delays surface to the caller. |
| 42 | +- `heygen_catalog_tts`: separate catalog IDs, `/v3/voices/speech`; explicitly choose |
| 43 | + `engine` (starfish, heygen_voice, elevenlabs). Native professional clone IDs do |
| 44 | + not belong here. Catalog output can be MP3 or WAV. Supply either suffix; the returned artifact |
| 45 | + path uses the actual response codec suffix and preserves the audio. |
| 46 | +- `heygen_voice_convert`: ACTIVE instant workspace voice only, at most300 seconds. |
| 47 | + Exactly one audio_path (<=16 MB), audio_url or asset_id. Output WAV. Supply a |
| 48 | + stable idempotency_key if resuming after an ambiguous timeout; no automatic retry. |
| 49 | +- `grok_tts`: xAI narration `/v1/tts`, not Grok Voice realtime. `language=auto` by |
| 50 | + default, voice_id defaults to eve; list_voices discovers voices. output_format |
| 51 | + is an object, e.g. {"codec":"wav","sample_rate":24000}; this file adapter exposes |
| 52 | + MP3/WAV, not headerless telephony codecs. Optional character alignment is saved. |
| 53 | +- `mistral_tts`: model_id `voxtral-mini-tts-3-0`; exactly one voice_id or local |
| 54 | + reference_audio_path. mp3/wav/flac/opus. Inline emotion/pace tags and expressiveness; |
| 55 | + free-form style instructions are not available at launch. No realtime TTS claim. |
| 56 | +- `mai_tts`: Azure Speech key/region, model_id MAI-Voice-2.1 or MAI-Voice-2.1-Flash. |
| 57 | + voice_id is a managed voice base such as en-US-Harper (the model suffix is added). |
| 58 | + The TTS selector translates speed and pitch through the Azure SSML adapter. |
| 59 | + This adapter does not provision personal voice clones or grant gated access. |
| 60 | +- `mistral_stt`: Voxtral Mini/Small Transcribe3, local audio files; timestamps are |
| 61 | + segments, not words. Diarization implies segment timestamps. Raw transcript JSON |
| 62 | + is persisted, including speaker and usage metadata. |
| 63 | +- `elevenlabs_soundtrack`: video_paths (<=10, combined <=200 MB / 600seconds), |
| 64 | + optional description/tags, add_sfx/use_audio_as_reference/separate_tracks. Separate |
| 65 | + tracks return playable files in a sibling *_tracks directory; never treat a ZIP |
| 66 | + as an audio file. The adapter rejects unsafe archive paths and oversized archives. |
| 67 | + |
| 68 | +## Images and video |
| 69 | + |
| 70 | +| Model | Explicit tools / hosts | |
| 71 | +|---|---| |
| 72 | +| Nano Banana2.1 | google_imagen, gemini_fal_image, gemini_replicate_image | |
| 73 | +| FLUX3 Image | bfl_image, flux3_fal_image, flux3_replicate_image | |
| 74 | +| FLUX3 Max video | flux3_video (fal) | |
| 75 | +| Vidu Q4 | vidu_video (fal), image_to_video or reference_to_video; no text-only route | |
| 76 | +| Fibo2 | bria_image direct or bria_fal_image | |
| 77 | +| MAI Image2.6 / Flash | mai_image, deployed Foundry models | |
| 78 | +| Grok Imagine1.5 / Lite | grok_video direct xAI | |
| 79 | + |
| 80 | +fal and Replicate have checked-in endpoint schemas under tools/provider_contracts. |
| 81 | +Pass provider_params only when documented by that host. Do not copy fal fields to |
| 82 | +Replicate or direct APIs. Existing Pro image model defaults remain unchanged on |
| 83 | +aggregators; select the new model explicitly. |
| 84 | + |
| 85 | +BFL bounding-box layouts are encoded in the prompt per BFL docs, not a fabricated |
| 86 | +`boxes` endpoint field. BFL supports up to10 reference images; width/height must be |
| 87 | +provided together. Its API reference is authoritative over older narrative examples. |
| 88 | +Bria persists structured metadata for later editing. Its shared endpoint can route |
| 89 | +some legacy combinations to older Fibo models; the direct tool rejects those |
| 90 | +combinations when advertising Fibo2. MAI requires model identity to match the |
| 91 | +configured deployment; PNG output and <=2,359,296 pixels with each side >=768. |
| 92 | + |
| 93 | +Grok1.5 references are limited to720p. Lite has no reference audio, reference images, |
| 94 | +last_frame or keyframes. Full1.5 supports preset voice references; custom audio |
| 95 | +references remain partner gated. Mid-video pins must be inside duration and occupy |
| 96 | +different1/3-second slots. No model substitution on failure. fal/Replicate/BFL/Grok |
| 97 | +jobs expose resume metadata; save job_path before polling and resume pending jobs |
| 98 | +instead of submitting duplicate paid generations. |
| 99 | + |
| 100 | +## Realtime is separate from narration |
| 101 | + |
| 102 | +`realtime_audio` is a bounded one-response file runner for Grok Voice3, |
| 103 | +Eleven Realtime Mini/Max and MAI-Transcribe-2-Streaming. Input is text or mono PCM16 |
| 104 | +WAV (16/24kHz; Eleven requires24kHz). Voice output is WAV; MAI output is event JSON. |
| 105 | +Input sending and event receiving run concurrently. MAI commits explicitly and |
| 106 | +waits for final transcription, not a commit acknowledgment. Eleven uses ephemeral |
| 107 | +session credentials and bursty input pacing for recorded files. |
| 108 | + |
| 109 | +For live audio, VAD, interruptions, multiple turns, or function calls, use the |
| 110 | +context-managed `tools.audio._realtime_session.RealtimeAudioSession` transport: |
| 111 | +`append_audio`, `commit`, `receive`, `request_response`, `cancel`, and |
| 112 | +`submit_tool_result`. The caller owns the session lifecycle and tool approvals. |
| 113 | +Received tool calls are never executed automatically. The finite file runner |
| 114 | +rejects tools because it cannot continue them after closing the socket. No microphone |
| 115 | +UI, browser app, or voice-agent orchestration is implied by this transport layer. |
| 116 | +Secrets remain in memory. No realtime translation or Kling4 support was added. |
| 117 | + |
| 118 | +## Primary references |
| 119 | + |
| 120 | +- https://ai.google.dev/gemini-api/docs/deprecations |
| 121 | +- https://elevenlabs.io/docs/api-reference/text-to-dialogue/convert |
| 122 | +- https://elevenlabs.io/docs/eleven-api/guides/how-to/websockets/realtime-tdd |
| 123 | +- https://elevenlabs.io/docs/api-reference/music/video-to-music |
| 124 | +- https://developers.heygen.com/docs/voices/heygen-voice-speech |
| 125 | +- https://developers.heygen.com/docs/voices/speech-to-speech |
| 126 | +- https://developers.heygen.com/reference/generate-speech |
| 127 | +- https://docs.x.ai/developers/model-capabilities/audio/text-to-speech |
| 128 | +- https://docs.x.ai/developers/model-capabilities/video/overview |
| 129 | +- https://docs.mistral.ai/studio/audio/overview |
| 130 | +- https://docs.bfl.ml/api-reference/utility/generate-an-image-with-flux-3 |
| 131 | +- https://docs.bria.ai/whats-new/fibo-2 |
| 132 | +- https://learn.microsoft.com/azure/ai-services/speech-service/mai-voices |
| 133 | +- https://learn.microsoft.com/azure/foundry/foundry-models/how-to/use-foundry-models-mai-image |
| 134 | +- https://learn.microsoft.com/azure/ai-services/speech-service/mai-transcribe-2-streaming-realtime |
| 135 | +- https://elevenlabs.io/docs/eleven-agents/guides/openai-realtime-compatibility |
| 136 | +- https://docs.x.ai/developers/rest-api-reference/inference/voice |
0 commit comments