Coding agents: skip this README and read AGENTS.md directly.
This is a community fork of Hyperflare, a local inference server for NVIDIA SM120/SM121 GPUs published by NVIDIA as a research prototype, formerly named Flare. It is not an official NVIDIA release. The text below this section is upstream's README as of the last merge, which still uses the name Flare.
This integration branch is upstream main (last merged on 2026-11-03, at
cc696a6b) plus the changes below, proposed upstream in four pull requests and
one issue, none merged yet. They speed up multi-host model preparation, and were
measured with DeepSeek V4.1 Flash on DGX Spark (GB10) hosts. Measurements are
from TP3 exports on three hosts, one run per configuration. Each row is measured
against the build it was developed on, not always upstream main; the linked
item gives the exact baseline. The code in this branch predates upstream's
renaming of packages to cores (bbad44fe); #124 and #125 have since been rebased
onto it, so those pull requests differ from the versions here.
| Change | Upstream | Phase | Before | After |
|---|---|---|---|---|
ignition join --local-checkpoint [require|prefer]: peers read their own copy of the checkpoint |
#126 (issue) | Load weights on the peers | 252 s | 44.7–50.1 s (per peer) |
--compile-headroom SIZE / [preparation] compile_headroom: host memory left free for compilers (default stays 8 GiB) |
#122 | Compile model forwards | 905.7 s | 507.7 s (6 GiB) |
| Link received payload objects into the cache instead of copying | #125 | Receive rank export payloads | 457.8 s | 280.6 s |
| Hash Engram planes on parallel readers | #157 | Assemble package payloads | 262.0 s | 169.1 s |
| Hash payload sources, and resolve shared payload cache objects, on parallel workers | #124 | Verify and publish shared data | 863.7 s | 426.9 s |
| The same, with the shared data already cached; Before is #124 without its cached-object verification | #124 | Verify and publish shared data (shared data already cached) | 238.3 s | 149.6 s |
The packages exported with #122, #124, #125 and #126 served normally with
flare run (262k-token retrieval 16/16).
#124 was measured again on top of upstream main at d2b5c5f3, which already
verifies cache payloads in concurrent batches, before being rebased onto
bbad44fe. One export per variant: "Export native packages"
took 2132 s on that main and 1692 s with #124 with an empty cache, and 1340 s
and 822 s with the previous export cached. See the pull request for the
per-phase figures and their limits.
A local prototype reduces the fixed prefill scratch of the DeepSeek 4.1 recipe,
which leaves more memory to the KV cache. It was written with Claude Opus 5.5 and
falls under upstream's contribution rules
on model optimizations, so it is not merged into this branch or proposed
upstream. Its two commits are in the branch prefill-scratch-overlap (on upstream
main at f571a550) and, on top of an earlier revision of this branch, in
prod-kv. Its measurement is
reported in #186.
Against upstream main at f571a550, TP3 on three GB10 hosts, three starts of
each package:
Upstream main |
With the prototype | |
|---|---|---|
| Fixed device memory per GPU | 96.71 GiB | 96.25 GiB |
| KV tokens, mean of 3 starts | 7.71M | 8.30M (+7.6 %) |
First-token top-20 log-probabilities were bit-identical for prompts of 836 to 94,341 tokens. Decode and prefill differences ranged from -2.9 % to +2.9 %, within the spread between starts of the same package (up to about 13 %); see the issue.
Flare is a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use. Its goal is to make the memory, bandwidth, and compute in workstations and compact systems such as DGX Spark go further, while keeping generation responsive.
This is a research prototype, and is made available via open-source to enable technical collaboration and collect early feedback.
The design rests on a few principles:
-
Embrace the standard PyTorch ecosystem at build time; serve from Rust. Ignition uses PyTorch, Transformers, and FlashInfer/b12x to load models, tune kernels, compile forwards, and capture CUDA graphs into a runtime package. Helios, the Rust runtime, serves that package without Python. Compilation and graph capture finish before serving.
-
Build on the model ecosystem. Reuse canonical model implementations wherever their dataflow fits, and specialize the parts that need different layouts or execution. Model development keeps the tools and libraries that make the Python ecosystem productive.
-
Make memory and state explicit. Plan buffers and execution capacities ahead of time. Treat conversation state—including recurrent state alongside KV caches—as a whole, so reuse, branching, and resuming preserve it correctly.
-
Keep the runtime small and shared. Model recipes own model computation; the runtime owns scheduling, cache storage, and request handling. A focused hardware and kernel stack keeps those boundaries manageable.
vLLM covers a much broader range of models, hardware, and deployment patterns. Flare trades that breadth for a focused serving contract around selected models and local workloads. Flare is not, and will never, be a replacement for production inference englines like vLLM.
Think of Flare as a specialized option when its hardware and model coverage fit your machine.
Start with model preparation and serving. See the CLI guide for configuration and the core design for the architectural reasoning.
Flare's project code is licensed under Apache 2.0. Contributions require the sign-off described in CONTRIBUTING.md. Third-party attributions are listed in NOTICE; upstream terms remain applicable to third-party code and assets.
The PyTorch-derived image resizing implementation in helios/ retains the
PyTorch BSD-style license, also distributed with
Heliopause. The DeepSeek recipe assets retain
their MIT license. The
heliopause/LICENSE and heliopause/NOTICE symlinks refer to the root files for
Python package distribution.
Upstream chat templates are fetched with the model checkpoints and annotated locally using bundled insertion patches. Gemma uses pinned Google sources. The custom Kimi segmented renderer is included as project code. Downloaded model assets and exported model packages retain their upstream terms; see frontend preparation.