Deliver streamed tokens live instead of after generation completes - #195
Conversation
|
@james-333i This one picked up a conflict after #196 went in as a squash. Would you rebase it onto |
streamResponse consumed an inner AsyncThrowingStream whose builder ran the entire generation loop synchronously on the consuming task, so every snapshot buffered and arrived in one burst after generation finished. Yield snapshots directly from the generation loop on the streaming task, and check for task cancellation between tokens so an abandoned stream stops decoding promptly.
d677bb5 to
4f6a869
Compare
There was a problem hiding this comment.
🟡 Changes recommended
A decoding failure from llama_decode is still silently swallowed (ending the stream as if successful), and should be surfaced as an error now that the generator is throws.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates streamResponse in LlamaLanguageModel so streamed output snapshots are yielded incrementally during generation (instead of being produced only after the full generation loop completes), improving real-time token delivery to consumers.
Changes:
- Replace the
generateTextStream-based loop with a directperformTextGeneration(..., onToken:)callback to yield snapshots as tokens are sampled. - Refactor
performTextGenerationto throw errors directly and emit tokens via anonTokenclosure. - Add a cooperative cancellation check inside the generation loop.
File summaries
| File | Description |
|---|---|
| Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift | Refactors streaming generation to yield per-token snapshots live during sampling rather than at completion. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| let decodeResult = llama_decode(context, batch) | ||
| guard decodeResult == 0 else { | ||
| break | ||
| } |
|
Merging now. Thanks, @james-333i! |
Brings in the merged huggingface#195, huggingface#205, huggingface#212 and huggingface#217 along with the follow-ups applied on merge (all-text prompt drop, boolean and number probe items). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
streamResponseran the whole generation loop before the consuming task received anything, so tokens arrived in one burst at the end. This yields each token as it is sampled.