Repository navigation
Export CUDA graph templates without holding executable instances - #245
Closed
voipmonitor wants to merge 1 commit into
Closed
voipmonitor wants to merge 1 commit into
voipmonitor wants to merge 1 commit into
Conversation
Large captured catalogues can exhaust device memory while instantiating graph executables that export does not need. Export needs the graph templates and their launch policy; native serving constructs its own executable instances. Retained capture records supported instantiation flags without instantiating executables. Export validates graph templates, buffers and context through a separate manifest; native replay still requires executable handles. Heliopause accepts exactly one flags source, either a live executable or explicit flags, and retains the existing serialized graph format. Tensor-parallel readiness and the native diagnostic use the corresponding resource contract. The change removes duplicate executable residency during preparation without changing serving arithmetic or core compatibility. On main 8c2e782, 42 focused CPU tests and nine SM120 GPU tests pass, covering template/executable validation, flags 0/1/8/9, and captured construction order. The implementation before the core/API rename also passed standalone native replay fixtures; full-model native qualification of this port is separate. Design-Signed-Off-By: Codex (GPT-6); design-review=cfcf
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preparing a large CUDA graph catalog can run out of device memory while instantiating executables that export never uses. Export needs graph templates and their instantiation flags; Helios creates its own executable instances when it loads a core. Avoiding the preparatory instances removes that duplicate residency without changing what serving executes.
Retained capture now records the supported instantiation flags and keeps the graph templates without creating executable handles. A separate export-manifest check validates templates, buffers and contexts; native replay still requires executables. Heliopause accepts exactly one source of flags: an existing executable or explicitly recorded flags. Non-retained capture keeps its existing executable path. The serialized core format and native launch policy are unchanged.
The implemented benefit is lower graph memory during export. This PR does not claim a serving-speed improvement or an isolated full-model memory measurement. Heliopause must be rebuilt for the Python API change; previously exported cores remain readable without re-export.
Validation on Flare main
8c2e782dwith this change, Torch 2.14.1+cu132 and SM120: 42 focused CPU tests and nine GPU tests passed. They cover template/executable validation, flags 0/1/8/9, tensor-parallel readiness and capture construction order. Four standalone fixed-address native replay cases also passed: both 64 GiB and 64 MiB shard caps, with Python exit or direct executable handover. Two GEMM fixture cases were blocked before export by a hardcoded CUDA kernel parameter-size assertion inhelios/tests/runtime_core_fixture.py:112. That fixture is byte-identical to main8c2e782d; its assertion precedesRuntimeCoreconstruction and export. These two cases do not establish native replay compatibility. The fixture subprocesses usedNVCC_PREPEND_FLAGS="-ccbin=/usr/bin/g++-13"to select the installed host compiler explicitly.