[voipmonitor]

Why

A reported Kimi K3.1 TP16 failure on a second long request motivated comparing export with retained CUDA executables against export that records template metadata and immediately resets the capture graph. Retention changes preparation-time resource lifetime. It does not transfer the Python executable into serving: Helios reconstructs and instantiates its own graphs when loading the core.

The supplied comparison does not yet isolate executable retention as the cause of recovery. That causal attribution remains unverified.

Changes

• record_export_graph instantiates and returns the original graph instead of copying template-only metadata and resetting it.
• The graph explicitly retains both external CUDA memory pools and caller-supplied kernel owners. Those references survive instantiation failure and remain until reset or collection. Tracked Python replay streams drain before release; native callers retain their existing explicit drain requirement. Owners must not reference the graph, which would create a finalizer retention cycle.
• Keep rejection of CUDA pools without an external owner and preserve collective consumer associations.
• Add weak-reference regressions for reset, collection and instantiation failure. Update export documentation and fixture names to describe retained handles accurately.

Result and compatibility

The external-owner lifetime fix is implemented and qualified by focused CPU tests. The global executable-retention path remains research-only as a remedy for the reported serving failure. It retains one executable per captured graph during preparation, increasing memory requirements relative to template-only recording; the total cost has not been measured here.

Serialized core format, model arithmetic and native serving policy are unchanged. This change does not repair an already exported core. No performance or model-quality improvement is claimed.

Validation

Review validation, without CUDA or end-to-end execution:

• Three ownership regression cases fail against submitted commit bb3ce1ea and pass with the fix. The original helper allowed both supplied pool owners and the kernel owner to be collected while the graph remained alive. The corrected helper retains them through release, drains registered replay streams first, and does not retain the graph itself.
• The focused graph/capture CPU suite passed 73 tests. The final metadata-only subset passed 12 tests. Ruff and git diff --check passed.
• Eight native metadata-validation tests fail identically on the submitted PR and its unchanged parent with the installed Heliopause extension, which does not match those tests' schema. They are not counted as passing validation.
• The CUDA export fixture checks handles, pool retention, manifests and serialized metadata. It does not replay a loaded core and therefore does not reproduce the second-request failure. It was not rerun during this review.

Run the ownership and metadata regressions from the repository root:

python -m pytest -q tests/preparation/test_capture_graph_metadata.py

Earlier report and attribution limit:

The original report described an illegal-address failure with template-reset export, four successful sequential long requests with retained executables, and successful C4 replay with a 128G host prefix cache. These serving results were not rerun during review.

The available preparation checkouts also differ in the eight-row paired gather: load_batch=2 if rows == 8 else 1 is present in the failing checkout and absent from the successful checkout. Their export logs report 3,980 kernels in 3,940 modules versus 3,940 kernels in 3,900 modules per rank. The corresponding cores are gb202-tp16-202611110058 and gb202-tp16-202611110504. These artifacts do not establish a single-change A/B comparison or prove that template reset caused the serving failure.

Before treating this as a serving correctness fix, isolate retention with otherwise identical preparation inputs and test repeated native replay. The caller-owner regression repaired here is independently demonstrated; it is not evidence of the cause of the original serving failure.

🤖 Generated with Claude Code

[voipmonitor]

Correction: a subsequent command/hash review found that this control was not an exact reconstruction of the original failing gb202-tp16-202611110058. The original export used load_batch=2 and native binary 8b7be5de6b7fd28f23e01abf21d0540990cf958e3372a6d3abb2ef348be239c6; this control used load_batch=1 and a different native binary. The result below is therefore only an A/B around 1108, not a conclusive reproduction of the original crash. I am rebuilding the original configuration and will replace this conclusion with the matched result.

Controlled native replay A/B, 2026-11-11

I tested the serving-causality question with two C4 packages built with the same FlashInfer sources, export arguments and native runtime. gb202-tp16-202611111352 restored only the graph-retention files to e774a468; gb202-tp16-202611111108 contains ec6190c0. Both exports contain 3,940 functions in 3,900 modules and share all 1,504 weight files.

The no-retention control completed long sequential requests with 831, 1,777, 2,896 and 4,110 prompt tokens, followed by four concurrent 1,024-token requests. It did not reproduce CUDA_ERROR_ILLEGAL_ADDRESS; the server remained healthy and host-cache store/fetch errors and disk records stayed zero. The retention package had already completed the same sequence and concurrent workload.

During the concurrent phase, the control completed 722 scheduler rounds and 4,092 emitted tokens (5.668 tokens/round, 30.868 ms/round). The retention package completed 745 rounds and 4,092 emitted tokens (5.493 tokens/round, 30.319 ms/round). Different adaptive acceptance makes these timings unsuitable for a performance claim.

Outcome: the original serving failure was not reproduced in this controlled comparison. Retention must not be presented as the proven fix for that illegal-address failure. The independently tested resource-ownership correction remains, but the original crash's cause is still unproven.

[voipmonitor]

Update: the original illegal-address failure has now been reproduced on Frank1. A no-retention C4 package built with the reconstructed paired-gather configuration and cold tuning failed on the original request-831 input after 0.38 seconds with cuEventQuery failed (700, CUDA_ERROR_ILLEGAL_ADDRESS), followed by HTTP 503. The native binary matches the original SHA-256 (8b7be5de6b7fd28f23e01abf21d0540990cf958e3372a6d3abb2ef348be239c6). A retention export using the same tuning cache and original development signer is in progress; its replay result will determine whether ec6190c prevents this exact failure.

[voipmonitor]

Final A/B update: a warm no-retention package (gb202-tp16-202611111441, 3,940 functions/3,900 modules) also completed the original request-831 input on the original native binary in 112.39 seconds with stop_reason=tool_use and a healthy server. By contrast, the cold no-retention package (gb202-tp16-202611111614, 4,030/3,990) reproduced CUDA_ERROR_ILLEGAL_ADDRESS in 0.38 seconds and HTTP 503. The retention package (gb202-tp16-202611111711, 3,945/3,905) completed request 831, the remaining long sequential requests and four concurrent 1024-token requests without an illegal address; its concurrent phase completed 639 scheduler rounds and 4,092 emitted tokens (6.404 tokens/round, 33.814 ms/round), with zero host-cache errors and zero disk records.

No-retention is therefore not sufficient to cause the failure, and retention is not necessary for the same request to succeed. The decisive variable demonstrated here is the graph/tuning variant, not graph retention. This serving-causality claim should be withdrawn from the PR. The independently tested resource-ownership correction may still have value, but the original illegal-address failure requires separate graph/tuning bisection rather than attribution to this PR.

[voipmonitor]

merged differently through another commit.