Repository navigation
Charge scheduler host handoff only where the queue drains - #182
Closed
voipmonitor wants to merge 1 commit into
Closed
voipmonitor wants to merge 1 commit into
voipmonitor wants to merge 1 commit into
Conversation
Graph costs added an exponential mean of wall time minus GPU time from every execution submitted to an empty queue, and queued executions never lowered it. With two outstanding submissions an ordinary decode or speculative round runs unqueued only when the host fell behind, so its excess measured host delay rather than the graph. On DeepSeek V4.1 TP4 at concurrency 32, one P128 round ran unqueued at 61 ms of wall time for 28.5 ms of GPU time and one P108 round at 54 ms for 27.2 ms. Their estimates rose from 29.0 to 34.5 ms and from 28.3 to 32.3 ms, and plain decode won the near-tie for more than 20 s. Queued rounds never lowered the handoff, and an unused shape was measured again only after expiry and when refresh credit reached it. Charge handoff only where it leaves the GPU idle: the median handoff of a graph's drained executions times the share of its last 16 ordinary executions that had to deliver completion feedback before the next submission. Prefill, state operations, grammar or host-stop lanes, copied proposals, and graphs with completion inputs keep their full handoff; pipelined rounds charge none. Refresh and acceptance probes still measure GPU time and drained handoff but do not change the share. GPU estimates, refresh credit, and policy are unchanged. DeepSeek V4.1 Flash TP4, switched GPUs 0-3 on frank1 and frank2, concurrency 32, seven drafts, verification budget at most 128, alpha 0.6, Engram tables in RAM, 30 s cells (tokens/s, mean of ten cells; cells with plain fraction above 0.3 in parentheses): host/package before after frank1 main 2445 (4) 2432 (0) frank1 B 2464 (4) 2483 (0) frank2 main 2370 (8) 2464 (0) frank2 B 2514 (0) 2497 (0) all 40 cells 2448 (16) 2469 (0) After the change every cell keeps P128 rounds at full depth. frank2 main package at concurrency 1, 8, and 16 (six cells each, mirrored order) shows no regression: 362 -> 374, 1339 -> 1357, and 1863 -> 1911 tokens/s. Service measurements apply the same change to 6fa01a4, the packages' export commit; the cost model there matches this base, whose later scheduler changes concern host-tier storage and reasoning budgets. Validation: Helios library tests (runtime-package and server features) pass except five frontend-template, package-cache, and image-hint tests that fail identically on main for missing local assets. New tests cover pipelined host delays, feedback-share charging, and probe exclusion; three existing assertions now expect the drained handoff of feedback steps and no handoff for MTP rounds. Limitations: a host that is systematically slower than the GPU leaves bubbles before pipelined rounds that neither model charges. Host delays themselves remain; this change only stops them from distorting graph costs. The P108/P128 choice follows acceptance estimates at alpha 0.6. Signed-off-by: Martin Vit <martin@voipmonitor.org> Design-Signed-Off-By: Claude (Opus 5.5); design-review=1c01 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
|
Merging a better fix for this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
At C32 (DeepSeek V4.1 Flash TP4, seven drafts, verification budget ≤ 128, alpha 0.6) the same binary and package swing between ~2300 and ~2600 tok/s from cell to cell. In the slow cells the scheduler sits in plain decode for tens of seconds.
A graph's cost is its median GPU time plus
handoff_ns, an exponential mean of wall − GPU time over executions submitted to an empty queue; queued executions never lower it. With two submissions in flight, a decode or speculative round runs unqueued only when the host fell behind, so the excess measures the host, not the graph. Example from unchanged main: one P128 round ran unqueued at 61 ms wall for 28.5 ms GPU and its estimate rose from 29.0 to 34.5 ms; plain decode then won the near-tie for over 20 s. The inflated estimate stays until it expires (30 s) and refresh credit reaches it. The host delays themselves come from acceptance-probe planning, fixed in #188.Change
Handoff is charged only where it leaves the GPU idle:
GPU estimates, refresh credit, and the selection policy are unchanged.
Results
C32, switched GPUs 0–3 on frank1 and frank2, Engram in RAM, prefix cache 64G, 30 s cells, before/after in mirrored order (B = clustered phased all-reduce package):
Every cell now runs full-depth P128 rounds. C1/C8/C16 (frank2, main): 362 → 374, 1339 → 1357, 1863 → 1911 tok/s, within noise. Service runs apply the same change to
6fa01a49, the packages' export commit (same cost model).Validation
docs/scheduling-runtime-design.mdandhelios/README.mddescribe the rule.Limitations
🤖 Generated with Claude Code