[javieratomic]

Related to #73 (a kernel-tuning MemoryError on GLM-5.3-Flash TP2), #242 (a multi-host export that stops when a host has no room for a forward compiler) and #270 (a DeepSeek V4.1 Flash TP3 export that stops in kernel tuning at 0572fd12); the #73 path was not tested on hardware, and the exports measured for the #242 and #270 conditions are summarized under Result.

Why

Forward compiler admission and kernel tuning leave a fixed 8 GiB of system memory free on each host. On a unified-memory GPU such as DGX Spark (GB10), the model's resident storage already occupies most of that memory: in a DeepSeek V4.1 Flash export with tensor parallelism 3 on three GB10 hosts (#242), two ranks had 12038 and 11939 MiB available after kernel tuning, with weights, KV cache and graph operands resident. Admission needs at least 4 GiB beyond the floor for one integrated compiler worker (the 2 GiB minimum host allowance plus 2 GiB of device storage), so a host with 12038 MiB available admits no worker and the export stops (#242). The right floor depends on the host and the model, so it should be a setting rather than a constant. Lowering it keeps the existing safety rule: on unified-memory GPUs, admission and tuning still leave free the greater of this floor and the preparation headroom (the recipe's preparation headroom plus [memory] reserve).

What changed

Before this change the floor was the private constant _HEADROOM_BYTES in ignition/preparation/host_memory.py. It is now COMPILE_HEADROOM_BYTES (still 8 GiB), the default of a compile_headroom_bytes argument in compiler_resources and CompilePool, which admit forward compiler workers, and in worker_limit, tuning_limits and candidate_scratch_limit, which size kernel tuning. The value comes from the new [preparation] compile_headroom setting (a size string such as "6G", default "8G") or the advanced --compile-headroom SIZE flag of export, debug-serve and bench, which overrides the file; negative, empty and unparsable values are usage errors. It is carried as Plan.compile_headroom_bytes through every recipe's PreparationConfig to the tuning session (including the nested session of Kimi K3.1 candidate-selection tuning) and ForwardCache, and multi-host peers receive rank zero's value, as they do for compile_workers. The plan dictionary, the "Select GPUs." line, the tuning report and the bench result record it. The low-memory threshold in release_unused keeps 8 GiB, and the 32 GiB per-worker concurrency estimate and single Inductor thread per worker are unchanged. docs/cli-design.md and docs/model-preparation.md document the setting and flag. With the default, admission and tuning are unchanged.

Result

Expected: on the host above, --compile-headroom 6G admits one compiler worker where the default admits none, and on hosts with more free memory a lower value can admit more workers under the 32 GiB estimate. A lower headroom is a per-host decision that trades the risk of exhausting host memory for being able to compile at all, or with more workers, which is why the default stays at 8 GiB.

Measured on bbad44fe, with an earlier head of this PR, on three GB10 hosts with DeepSeek V4.1 Flash, tensor parallelism 3 and one rank per host (details): with the default, bbad44fe stopped at admission in one export, 713 MiB short of the 12288 MiB minimum on rank 0's host, and passed in another by under 0.3 GiB. bf5ca911 (this change on bbad44fe) with --compile-headroom 6G admitted one compiler per host and compiled 38/38 forwards in 1484.4 s, against 1488.6 s in the default export that passed. In that 6 GiB export all three hosts were also above the default minimum at admission, so it does not show a case that only 6 GiB admits; 6 GiB lowers the admission minimum by 2 GiB. During its kernel tuning, swap use on rank 0's host rose by about 2.5 GiB; the exports do not show whether the setting caused it.

Measured on 0572fd12 and on the current head, same hosts and model (#270): with recipe defaults, 0572fd12 stops 17.3 to 18.9 s into kernel tuning with a 0-byte scratch limit, because the hosts have 7314 to 8083 MiB available before the scratch reservation, below the 8 GiB floor. f25a0386 (this PR) with --compile-headroom 4G completed the export in 9527 s: kernel tuning took 4611.9 s and all 38 forwards compiled in 1848.1 s with one compiler worker per host. With 6 GiB, the 7256 to 7483 MiB that rank 0's host had before the reservation leave at most 315 MiB for the scratch limit before the pending bytes are subtracted; that export was not run. The 4 GiB export is a way past #270, not a fix for it: during kernel tuning rank 0's host fell to 3381 MiB available and its swap use grew from 628 to 4456 MiB. The core that export wrote was not served: the runtime of 0572fd12 rejects its RoCE proxy ABI (#271).

Evidence

CPU tests on 0572fd12 plus this commit (CUDA hidden, environment from uv.lock): tests/cli, tests/preparation, tests/recipes/test_declarations.py, tests/recipes/test_minimax_m3.py and tests/recipes/muse_spark gave 2294 passed, 70 failed and 248 skipped. 0572fd12 alone gives 2249 passed, 248 skipped and the same 70 failures in the same environment; they are pre-existing, mostly in the native recipe contract (21) and chat-template patch (15) tests. New tests cover setting and flag parsing and validation, the plan and startup line, every recipe's PreparationConfig, worker_limit, tuning_limits, candidate_scratch_limit and the tuning report, and CompilePool admission: with 12038 MiB available, the default admits no worker and reports the headroom, 6 GiB admits one, and with 72 GiB available 4 GiB admits two workers where the default admits one. GPU tests were not run. The hardware exports on bbad44fe, with their memory samples, are in this comment; those on 0572fd12 are in #270.

🤖 Generated with Claude Code

[original-el8]

@javieratomic GitHub reports that this PR has merge conflicts with main. A local git merge-tree check also confirms conflicts for PR head 2ca9cfa49e60 against main at c65024dba32b (2026-11-05).

Conflicting files:

• ignition/cli/config.py
• ignition/recipes/qwen38/bootstrap.py
• tests/cli/test_serve.py

Please update the branch against main (rebase or merge), resolve these conflicts, and rerun the affected checks.

[javieratomic]

@original-el8 Rebased onto c65024db. The conflicts were main's new force_moe_decode_a16 setting and tests, the max_age prefix-cache setting, and the DFlash2 snapshot check in the Qwen3.8 bootstrap; both sides are kept. While checking the rebase I found that bench accepted --compile-headroom but did not pass it to the plan; it does now, with a test. The CPU tests show the same failures as on c65024db, apart from one CompilePool test that needs 10 GiB of free host memory and passes when it is free. The commit message and the description give the counts.

[original-el8]

@javieratomic GitHub reports merge conflicts with main; local git merge-tree confirms them for PR head 57b4de578490 against main at 91cfb3695f10. Conflicting files:

• ignition/preparation/aot.py
• ignition/preparation/host_memory.py
• ignition/preparation/tuning.py

Please reconcile these files with main and rerun the checks affected by the resolution.

[javieratomic]

Reworked onto bbad44fe. Since c8d4c32 admits forward compiler workers per host with a 16 GiB estimate and one Inductor thread each, the setting now applies to that admission (compiler_resources) and, as before, to kernel tuning (worker_limit, tuning_limits, candidate_scratch_limit, including Kimi K3.1's candidate-selection session). The earlier measurements in this PR (about 510 s against 905.7 s for "Compile model forwards", and the serving checks of that export) came from extra Inductor threads per compiler, a mechanism main no longer has, so they are withdrawn; this version has not been measured on hardware. On a GB10 rank with 12038 MiB available, as in #242, the default admits no compiler and --compile-headroom 6G admits one (covered by the new CompilePool tests). CPU tests: 2198 passed, 70 failed, 238 skipped on this branch against 2153 passed and the same 70 failures on bbad44fe.

[javieratomic]

Hardware measurements of bf5ca911 against bbad44fe, on three DGX Spark GB10 hosts (DeepSeek V4.1 Flash, tensor parallelism 3, one rank per host). Each export was stopped after "Compile model forwards". The first two started from copies of the same partially tuned kernel cache and each spent about 4270 s tuning kernels; the third started from the cache the second one produced and measured none.

Tree	Compile headroom	Prepare and tune kernels	Compile model forwards
bbad44fe	8 GiB (default)	4268.2 s	fails after 2.5 s
bf5ca911 (this PR)	--compile-headroom 6G	4271.5 s	38/38 in 1484.4 s; 3 compiler workers, per rank [1, 1, 1]
bbad44fe	8 GiB (default)	404.2 s, 0 measured	38/38 in 1488.6 s; 3 compiler workers, per rank [1, 1, 1]

The first export stopped with MemoryError: host '…' has 38 missing forwards but no capacity for a forward compiler (11575 MiB available host memory, 8192 MiB headroom) on rank 0's host, 713 MiB short of the 12288 MiB minimum for one compiler under the default (8192 MiB headroom plus a 4096 MiB single-worker allowance); with 6 GiB that minimum is 10240 MiB. The second export compiled on all three hosts, but it does not show that 6 GiB was needed: the 2 s samples between the end of kernel tuning and the start of the compiler process read 13919 to 13986 MiB of MemAvailable on rank 0's host and 12579 to 12702 MiB on the other two, all above 12288 MiB. In that export about 2.5 GiB on rank 0's host had moved to swap during kernel tuning, close to the 2.3 GiB by which its MemAvailable at admission exceeded the failing export's 11575 MiB; the other two hosts, whose swap use did not change, read about the same in both exports (12569 to 12688 MiB in the failing one). In the third export the default passed with 12563 to 12586 MiB on rank 0's host, under 0.3 GiB above the minimum. In the two exports with the default, it admitted or rejected a compiler on rank 0's host by 0.3 to 0.7 GiB; 6 GiB adds 2 GiB of margin.

Memory was sampled every 2 s on each host. In the two exports that compiled, each compiler worker peaked at 4771 to 4995 MiB resident (VmHWM) and the Inductor compile_worker process that starts it at 631 to 656 MiB (PSS). During "Compile model forwards" the lowest MemAvailable on any host was 7547 MiB with 6 GiB and 7654 MiB with the default, and swap use did not grow on any host. During kernel tuning in the 6 GiB export, swap use on rank 0's host rose from 374 MiB to about 3000 MiB (MemAvailable minimum 7018 MiB) and stayed there through compilation; it did not rise during tuning in the two default exports, and it was constant on the other two hosts in all three. With 6 GiB the setting also leaves 2 GiB less headroom to kernel tuning (worker_limit, tuning_limits, candidate_scratch_limit), and rank 0's host also runs processes outside the export, so these runs do not show whether the swap rise came from the setting.

[original-el8]

@javieratomic GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head f25a03863440 against main 97cfeb0001f1. Conflicting paths:

• ignition/cli/bench.py

Please reconcile the branch with main and rerun the checks affected by the resolution.