Repository navigation
Commit 44ec239
Remove repeated work from scheduler planning (#188)
* Remove repeated work from scheduler planning
Funded acceptance learning compared every probe candidate with the ordinary
winner slot by slot, and each comparison of a packed round re-estimated the
whole round's prefix lengths, including lane values, while the allocation
loop re-evaluated every lane's acceptance curve for each extra row. At
concurrency 32 one plan call took 10-35 ms whenever acceptance credit was
full. The worker does not poll completions while it plans, so a finished
round waited for the plan before its successor or a draining probe could
start: on DeepSeek V4.1 TP4 every few seconds, and for about 0.7 s after 32
requests arrived.
Planning also repeated linear work per lane and per shape:
- Each speculative-token bound scanned every lane of the submitted steps to
find the request's pending advance. The ceiling is now tracked on the
request where commit and completion already track its pending count.
- Packed rounds rebuilt rankings, gains, and per-lane curves for every shape
and priorities for every ordering. Shapes sharing draft and width bounds now
share one ranking, lane curves are evaluated once per call, priorities once
per shape, and only the leading cohort of each ordering is sorted.
- Padding pruning compared all candidate pairs and re-estimated round lengths
for each pair. Only candidates with the same kind, cohort, and prefill row
are compared, and each candidate's lengths are estimated once.
- Service rates and next-progress values evaluated the forecast curve for
every batch and width, and every speculative duration repeated a hashed
lookup of the full graph, which the packed round table already includes.
Selection, round lengths, and their arithmetic are unchanged. Seeded
scheduler traces over 1056 scenarios, with 1-64 requests, host and device
prefix allocation, fixed widths, mixed graphs, grammar and host-stop lanes,
queued rounds, retirements, and timing and acceptance probes, record every
plan, lane value, pruned candidate list, probe choice, and round length:
about 210 000 plans compare byte for byte with main; the first quarter also
in a debug build.
Median plan time with one round queued (release test build):
requests allocation main after acceptance probe: main after
8 device 140 us 60 us 2563 us 111 us
16 device 289 us 96 us 2153 us 201 us
32 device 669 us 196 us 5312 us 413 us
64 device 1773 us 436 us 13380 us 943 us
32 host 754 us 250 us 1363 us 497 us
64 host 1957 us 615 us 3546 us 1215 us
DeepSeek V4.1 Flash TP4 on switched GPUs 0-3 of frank1 and frank2,
concurrency 32, seven drafts, verification budget at most 128, alpha 0.6,
Engram tables in RAM, with the exposed-handoff cost change in both builds
(tokens/s, mean of ten 30 s cells, worst cell in parentheses):
host/package before after
frank1 main 2471 (2428) 2524 (2490)
frank1 B 2474 (2432) 2529 (2495)
frank2 main 2478 (2398) 2537 (2503)
frank2 B 2450 (2313) 2540 (2515)
all 40 cells 2468 2532 (+2.6%)
In a diagnostic run of the same service, the GPU busy share of the steady
window rose from 98.4-98.9% to 99.8-99.9%, and no worker iteration exceeded
10 ms.
Validation: Helios library tests (runtime-package and server features) pass
in release and debug builds except five frontend-template, package-cache,
and image-hint tests that fail identically on main for missing local assets.
Debug builds check the tracked pending ceiling against the submitted lanes on
every use. A new test covers queued rounds, completion, and retirement.
Limitations: while a winning probe waits for the queue to drain, the worker
still plans again on every iteration; each plan is now short.
Signed-off-by: Martin Vit <martin@voipmonitor.org>
Design-Signed-Off-By: Claude (Opus 5.5); design-review=1c01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Reuse scheduling cursors and rank speculative prefixes
Derive pending advance from the scheduled and consumed cursors, retaining
submission scans for completion reconciliation. Stream acceptance candidates
into winner selection without storing all candidate length arrays.
Rank monotone prefix survival once with depth and slot tie-breaking. Share
the ranking with host cohort allocation and preserve request limits, fixed
widths, and GPU-owned allocation ceilings.
Validation: 346 scheduler and runtime phase tests pass in both debug and
release builds; two tests are ignored by default. Differential tests cover
host/GPU allocation, MTP/copy proposals, ties, tails, and cohort growth. A
release CPU benchmark measures planning with one queued round and funded
or unfunded acceptance probes. GPU serving measurements were not run.
Design-Signed-Off-By: Codex (GPT-6); design-review=1c01
Signed-off-by: Luke Alonso <lalonso@gmail.com>
---------
Signed-off-by: Martin Vit <martin@voipmonitor.org>
Signed-off-by: Luke Alonso <lalonso@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Luke Alonso <lalonso@gmail.com>1 parent cc696a6 commit 44ec239
7 files changed
Lines changed: 391 additions & 80 deletions
File tree
- helios/src
- scheduler
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2087 | 2087 | | |
2088 | 2088 | | |
2089 | 2089 | | |
2090 | | - | |
| 2090 | + | |
2091 | 2091 | | |
2092 | 2092 | | |
2093 | 2093 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
27 | 31 | | |
28 | | - | |
29 | | - | |
30 | | - | |
| 32 | + | |
| 33 | + | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
34 | | - | |
| 37 | + | |
35 | 38 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
46 | 50 | | |
47 | | - | |
48 | | - | |
| 51 | + | |
| 52 | + | |
49 | 53 | | |
50 | 54 | | |
51 | 55 | | |
52 | | - | |
| 56 | + | |
53 | 57 | | |
54 | 58 | | |
55 | | - | |
| 59 | + | |
56 | 60 | | |
57 | 61 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
0 commit comments