Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions docs/TODO.txt
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,6 @@
[ ] Qualify prompt scoring on remaining recipes and cache lifecycles;
see docs/logprobs-design.md for TP4 costs and numerical limits.

[ ] Respect ignore_eos when classifying EOS at the output limit;
see helios/src/server/runtime.rs.

[ ] Validate embedding readiness timestamps in full Qwen3.8 PLE serving and
cross-node RoCEnante TP packages for Qwen3.8 and DeepSeek V4.1.

Expand Down
19 changes: 19 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,25 @@ or partial tags. Structured answers become verbatim when ordinary answer content
begins, preserving scalar JSON, Lark output, and markup inside string values.
Shared runtime handling owns token accounting, stop filtering, and termination.

The terminal event carries a termination cause, and each protocol maps the
cause to its finish reason. An output limit reports `length` on Chat
Completions and Completions, `max_tokens` on Messages, and `incomplete` with
`incomplete_details.reason` `max_output_tokens` on Responses, even when parsed
calls completed the answer. Otherwise, when parsed calls complete an answer the
user did not stop, Chat Completions reports `tool_calls` and Messages
`tool_use`; raw Completions rejects tool parameters. Any other cause reports
`stop` on Chat Completions and Completions, `stop_sequence` on Messages for a
matched string stop and `end_turn` for every other cause, including an honored
end-of-sequence token and an explicit token stop, and `completed` on Responses.
The cause is decided at the terminal token: an honored end-of-sequence token or
a matched string stop that lands on the final allowed token is the cause, so
the response reports the stop reason rather than the limit reason. A request
admitted with `ignore_eos` does not treat end-of-sequence tokens as
terminating, so one that exhausts its output limit reports the limit reason
even when its last generated token is end-of-sequence. A request the scheduler
cannot grow past page capacity also reports the limit reason before the
declared limit is reached.

A reasoning delimiter that the tokenizer encodes as one token, such as
`<think>`, `</think>`, `</mm:think>`, or Gemma 4's `<channel|>`, opens or
closes reasoning only when the model generates that token. When a model writes
Expand Down
12 changes: 4 additions & 8 deletions helios/src/server/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1844,15 +1844,11 @@ impl<'a> Worker<'a> {
None => GenerationEvent::Token(token),
});
}
let reason = if user_stop
|| request
let eos = !request.incoming.prompt.ignore_eos
&& request
.last_token
.is_some_and(|token| self.config.eos_token_ids.contains(&token))
{
"stop"
} else {
"length"
};
.is_some_and(|token| self.config.eos_token_ids.contains(&token));
let reason = if user_stop || eos { "stop" } else { "length" };
request
.incoming
.emit(GenerationEvent::Finished {
Expand Down
33 changes: 33 additions & 0 deletions helios/src/server/runtime/phase_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1424,6 +1424,39 @@ fn exact_system_and_prompt_checkpoints_publish_only_after_snapshot_completion()
assert_eq!((metrics.prefill_tokens, metrics.generated_tokens), (13, 3));
}

#[test]
fn ignored_eos_at_the_output_limit_finishes_for_length() {
let fixture = Fixture::new();
let (sender, receiver) = mpsc::channel();
let (mut incoming, mut generation) = fixture.incoming(vec![0], 2);
incoming.prompt.ignore_eos = true;
sender.send(vec![incoming]).unwrap();
let mut worker = fixture.worker(receiver);
assert_eq!(worker.admit().unwrap(), ControlFlow::Continue(true));
advance(&mut worker, GraphKind::Init, 1, None);
advance(&mut worker, GraphKind::Prefill, 2, Some(5));
advance(&mut worker, GraphKind::Snapshot, 3, None);
advance(&mut worker, GraphKind::Decode, 4, Some(5));
advance(&mut worker, GraphKind::Snapshot, 5, None);
assert!(worker.publish().unwrap());
let mut finished = Vec::new();
while let Ok(event) = generation.events.try_recv() {
if let GenerationEvent::Finished {
generated_tokens,
reason,
termination,
..
} = event
{
finished.push((generated_tokens, reason, termination));
}
}
assert_eq!(
finished,
vec![(2, "length", crate::output::Termination::OutputLimit)]
);
}

#[test]
fn disabled_host_store_preserves_decode_prefix_reuse_and_cancellation() {
let fixture = Fixture::new();
Expand Down