[brandonmmusic-max]

The preparation dependency named one exact local-inference-lab/flashinfer
commit in pyproject.toml and uv.lock; updating FlashInfer required changing
the revision in pyproject.toml and refreshing uv.lock before preparation
could use it. scripts/setup_preparation.py ran
uv sync --upgrade-package flashinfer-python, but that command could not move
past an exact pin. The documented setup commands used --locked, which
installed the recorded commit and did not resolve main.

The dependency is declared as
flashinfer-python[b12x] @ git+https://github.com/local-inference-lab/flashinfer.git@main,
and [tool.uv] upgrade-package = ["flashinfer-python"] makes plain uv lock,
uv sync, and uv run resolve main's head commit and record it in uv.lock.
.github/workflows/flashinfer-main.yml proves that resolution, the build, the
install, and the imports on a GPU-less runner. No person or bot edits a pin. The
Rust Flare binary does not link FlashInfer, so a FlashInfer commit change does
not require rebuilding it; a kernel change requires a re-export.

What changed, path by path

Path	Before	After
Fresh clone, documented setup (uv sync --extra prepare --extra test)	Installed the exact revision in pyproject.toml and uv.lock.	Resolves main's head commit, installs it, and rewrites the FlashInfer entry in uv.lock.
Existing checkout, plain uv sync or uv run	Stayed at the exact revision.	Resolves main's head commit and rewrites uv.lock.
scripts/setup_preparation.py	Its uv sync --upgrade-package flashinfer-python could not move past the exact revision.	Resolves main's head commit, rewrites uv.lock, and rebuilds FlashInfer and Heliopause.
.venv/bin/ignition or an activated ignition	Used the installed revision.	Uses the last synced commit. Run uv sync or scripts/setup_preparation.py first when a preparation must use main's head.
uv sync --frozen	Installed the recorded commit.	Installs the recorded commit and does not re-resolve.
uv sync --frozen --offline	Not documented.	Installs the recorded commit without network access when the uv cache holds the packages.
uv sync --offline without --frozen	Not documented.	Fetches main and fails when the network is disabled. It is not a tracking path.
Multi-host peers	Every host used the same committed lock.	The first host resolves main and records it in uv.lock; peers copy that file and run uv sync --frozen. Every host can confirm its installed commit from direct_url.json.
CI	No normal job installed or used FlashInfer.	.github/workflows/flashinfer-main.yml runs on every pull request without a path filter, on pushes to main, daily, and by manual dispatch. It first removes the runner's preinstalled Android, .NET, GHC, and CodeQL toolchains, which the job does not use, and points every uv command at one cache, so the FlashInfer checkout (about 1 GB of source at ea74ea61) is fetched once. It then reads the exact refs/heads/main hash, runs plain uv lock, checks that the requirement is @main with upgrade-package, fails unless the resolved commit equals that hash, installs that commit in a standalone venv, checks direct_url.json and flashinfer.__git_commit__, and imports flashinfer, b12x, and the b12x.comm.pcie startup-tuning APIs with no visible GPU.
Docker and packaging builds	No FlashInfer install.	Unchanged.
flare run with an exported package	Uses captured kernels and graphs.	Unchanged. A FlashInfer kernel change requires re-export.

The docs state which commands track main (uv lock, uv sync, uv run,
scripts/setup_preparation.py) and which use the last synced commit
(.venv/bin/ignition, an activated environment, --frozen, --no-sync). They
also state that plain uv commands rewrite uv.lock, that the rewritten
FlashInfer entry should be committed only with a change that needs that commit,
and that --offline must be paired with --frozen. The Muse Spark section
states that the commit recorded in uv.lock lacks the required operators and
uses uv sync --frozen --extra prepare --extra test to restore the recorded
dependency. Muse Spark itself was not exercised.

What this does not do

• It does not make .venv/bin/ignition or an activated environment fetch a new
  commit. Those paths use the environment that was last synced. The update step
  is uv sync or scripts/setup_preparation.py.
• It does not make flare run or an exported package update. Serving uses the
  graphs and kernels captured at export.
• It does not install Flare's full preparation environment in CI. The CI job
  installs FlashInfer with FlashInfer's own resolved dependencies. It proves
  the FlashInfer commit builds, installs, and imports, not that the complete
  CUDA 13.2, torch 2.14.1, Heliopause preparation environment builds.
• It does not guarantee that a FlashInfer main change preserves GPU kernel
  correctness. The workflow catches resolution, build, install, and import
  failures.
• It does not exercise Muse Spark or fix the pre-existing Windows native test
  failures. It corrects only the Muse Spark section's dependency wording.
• It does not add a runtime update path to serving. Serving stays inside the
  prepared-graph boundary.

Validation

Environment for the resolution and CI checks: uv 0.11.30, Python 3.12.3,
Linux, no GPU work.

On this branch merged with main at d2b5c5f3 (merge commit 36fd8d19), with
FlashInfer main at ea74ea610ce89d4499ad4b1ecb6c9b525fc2a795:

• git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/main
  returned ea74ea610ce89d4499ad4b1ecb6c9b525fc2a795. Plain uv lock recorded
  that commit, uv lock --check exited 0, and the workflow's declaration and
  lock assertions passed with the locked commit equal to that hash.
• The workflow's install-and-import step, run locally in a fresh Python 3.12
  venv with FLASHINFER_BUILD_NO_PIP=1 and BUILD_NVEP=0, installed FlashInfer
  0.7.3 at ea74ea61. direct_url.json and flashinfer.__git_commit__
  reported that commit, and flashinfer, b12x, and the b12x.comm.pcie
  startup-tuning APIs imported with CUDA_VISIBLE_DEVICES empty and
  torch.cuda.is_available() false. The local run used PyPI plus the PyTorch
  cu132 index and resolved torch 2.15.0+cu132; the workflow uses PyPI only.
• pytest tests/preparation/test_setup_preparation.py tests/preparation/test_packaging.py
  reports 6 passed and 1 failed. The static declaration test
  (test_flashinfer_declaration_tracks_main_statically) passes; it is a
  declaration check, and the workflow is the behavior proof. The failure,
  test_server_cli_help_does_not_prepare_model, fails identically on main at
  d2b5c5f3: python -m ignition --help prints debug-serve on a line by
  itself, so the expected "\n debug-serve " text is absent.
• GitHub Actions on a79c43cc in brandonmmusic-max/flare-review, which adds
  the workflow's disk and cache step to 36fd8d19: the FlashInfer main job
  passed in 5m16s (run 42553853087). It locked ea74ea61, installed
  FlashInfer 0.7.3 at that commit, and passed every assertion. In Native
  serving (run 42553853071), the rdma job passed, the Ubuntu job failed one
  test (legacy_templates_receive_image_blocks_and_return_original_sources),
  and the Windows job failed eight hub and load-package tests with 149 passed.
  Native serving on main at d2b5c5f3 (run 42544646868 in
  NVIDIA/spark-local-inference) fails the same Ubuntu test and the same eight
  Windows tests, at the same steps, and its rdma job passes.

On commits 3ce4caf3 and 83c49b02, which differ only in comments and
documentation, without main's d2b5c5f3 changes:

• git ls-remote https://github.com/local-inference-lab/flashinfer.git refs/heads/main
  returned 1f8ed8a3c46e27d03cff54e08f23e9bc50476d57.
• In a scratch copy of this tree, plain uv lock resolved flashinfer-python
  to that commit with source ?rev=main#1f8ed8a3.... The resolved commit
  equaled the remote hash, and uv lock --check passed.
• A lock recording ddbe93fc moved to 1f8ed8a3 under plain uv lock, plain
  uv sync --extra api-test, and plain uv run. uv run --no-sync left the
  lock unchanged. uv sync --frozen --extra prepare --extra test --dry-run
  reproduced the recorded commit, and uv sync --frozen --offline --extra prepare --extra test --dry-run did too. Plain uv sync --offline --extra api-test failed to fetch main, which is why the offline form uses
  --frozen.
• An exact ddbe93fc pin without upgrade-package stayed at ddbe93fc under
  plain uv lock, and the workflow's comparison failed as intended. The
  workflow does not pass with a stale exact pin.
• A real uv pip install "flashinfer-python[b12x] @ git+...@1f8ed8a3..." in a
  fresh venv built and installed. direct_url.json and
  flashinfer.__git_commit__ reported 1f8ed8a3. import flashinfer,
  import b12x, and the b12x.comm.pcie startup-tuning imports succeeded with
  CUDA_VISIBLE_DEVICES empty and torch.cuda.is_available() false. The cold
  install took 4m22s and the venv was 6.4 GB. A warm install took 2s.
• The workflow YAML parses with PyYAML, and the extracted run scripts pass
  bash -n. actionlint was not available.
• End-to-end check of 83c49b02. On one RTX PRO 6000 (SM120), the tree was
  built with the documented uv sync --extra prepare --extra test. That
  installed FlashInfer 1f8ed8a3c46e27d03cff54e08f23e9bc50476d57, the head of
  local-inference-lab/flashinfer main at build time, with no pin edited.
  Qwen3.8-27B NVFP4 with the DFlash2 drafter was exported (rc 0) and served
  with flare run. A prefill-only benchmark cell on a fresh server returned
  rc 0 with 0 device errors. The functional checks passed 6/6. The benchmark
  script (decode and three prefill cells) returned rc 0 with 0 device errors.
  The Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure) #116 reproduction at --memory-limit 50G had
  no failure in 143 prompts and 0 device errors. This exercises the documented
  build path once, at the main commit that was head at build time. A build of
  this tree with plain uv sync --extra prepare --extra test resolved
  local-inference-lab/flashinfer main at
  42b221db437e80fc66e41f5e630da88aefc387e7, rewrote the uv.lock FlashInfer
  entry from 1f8ed8a3c46e27d03cff54e08f23e9bc50476d57 to that commit, built
  FlashInfer, and installed it; the distribution's direct_url.json reported
  42b221db437e80fc66e41f5e630da88aefc387e7. No pin was edited.

Limitations

• CI scope. The CI job uses FlashInfer's own resolved dependencies, not
  Flare's pinned torch 2.14.1+cu132. It sets FLASHINFER_BUILD_NO_PIP=1 and
  BUILD_NVEP=0, matching scripts/setup_preparation.py, so the build hook
  does not run nested installs or build the NIXL and NCCL EP backends. It does
  not build Heliopause or run GPU preparation. The full uv sync --extra prepare --extra test was not run in CI. It was run once on one RTX PRO 6000
  for the end-to-end check above.
• CI size and time. At FlashInfer ea74ea61 the standalone install
  measures 6.9 GB locally, and the FlashInfer source tree is 1.01 GB in 19,374
  files (0.45 GB in 9,281 files at 1f8ed8a3). Without the disk step, the
  hosted runner ran out of space extracting torch (GitHub Actions run
  42553335278 in brandonmmusic-max/flare-review, on 36fd8d19). With the
  disk step, the runner had 14 GB free before the removal and 32 GB after it,
  the job used about 12 GB, and it passed in 5m16s (run 42553853087, commit
  a79c43cc). At 1f8ed8a3 the install was about 6.4 GB and took 4m22s cold locally, and
  the job passed in 3m28s (run 42056263565, commit 83c49b02). If the hosted
  runner runs out of disk or time, the job fails closed.
• Re-export. Existing packages keep their prepared kernels. A FlashInfer
  kernel change requires an export. The Rust flare binary does not link
  FlashInfer, so it does not need rebuilding for a FlashInfer commit change.
• Modified lock. Plain uv commands rewrite the FlashInfer entry in
  uv.lock. Commit the rewritten entry only with a change that needs that
  FlashInfer commit. A uv.lock that records a commit, with uv sync --frozen,
  reproduces the environment it recorded while that commit remains fetchable
  (see FlashInfer main history below).
• Offline. Offline preparation was validated with --dry-run only. A full
  offline prepare install needs a populated uv cache and was not run.
• Main can break Flare. A FlashInfer main change that breaks Flare's use of
  b12x reaches the next build. The workflow catches resolution, build, install,
  and import failures, not GPU kernel correctness. A lock that records a
  working commit, with --frozen, reproduces that environment while that
  commit remains fetchable.
• FlashInfer main history. The history of local-inference-lab/flashinfer
  main at ea74ea61 contains neither 1f8ed8a3, the commit the end-to-end
  check used, nor 42b221db, the commit the later build resolved. GitHub's
  comparison of ea74ea61 with 1f8ed8a3
  reports diverged histories, with 2,408 commits only in ea74ea61 and 247
  only in 1f8ed8a3. GitHub still serves 1f8ed8a3 by hash. Reproducing a
  report from a uv.lock that records a commit later removed from main depends
  on GitHub retaining that commit.
• Muse Spark. The Muse Spark section's dependency wording states that the
  commit recorded in uv.lock lacks the required operators and that
  uv sync --frozen --extra prepare --extra test restores the recorded
  dependency. Muse Spark itself was not exercised.
• Native CI. The Native serving workflow fails on main itself. On main
  at d2b5c5f3 (run 42544646868), the Ubuntu job fails
  legacy_templates_receive_image_blocks_and_return_original_sources and the
  Windows job fails eight hub and load-package Rust tests. On a79c43cc (run
  42553853071) the two jobs fail exactly the same tests at the same steps,
  and the rdma job passes on both. The same eight Windows tests also failed on
  83c49b02 (run 42056263704) and on main at 0009b89d (run 41941475176)
  and 78b18b1c (run 41885630157). This change touches no Rust and does not
  address them.
• GPU coverage. The CI jobs do not run GPU preparation. The end-to-end
  check above covers one model on one RTX PRO 6000 at FlashInfer 1f8ed8a3,
  plus a build at 42b221db. No GPU run covers ea74ea61, the commit
  uv.lock records. This change does not modify kernels, graphs, or serving,
  and the dependency resolution behavior is covered by the CPU checks above.

I prepared this PR with ai assistance and remain responsible for the outputs. I have reviewed all of the code and conducted all testing and validation referenced herein. BMM

[original-el8]

This changes preparation's dependency policy: ordinary uv lock, uv sync, and uv run can select a different FlashInfer commit and rewrite the lockfile for the same Flare checkout. --frozen preserves the recorded environment. Categorized as triage:design-review / area:preparation so the automatic-update default receives an explicit maintainer decision.

The added CI checks dependency resolution, standalone installation, and imports. It does not qualify Flare's complete preparation environment or GPU behavior against each selected commit; the reported SM120 export qualifies the recorded combination only. The decision should cover reproducible bug reports, using one lock across TP peers, and recovery from an incompatible dependency update. This PR is distinct from an ordinary pin update and does not by itself resolve reports requiring particular kernel fixes and re-export.

[brandonmmusic-max]

Thanks. On the three points:

• Reproducible bug reports: uv.lock still records the exact FlashInfer commit, and uv sync --frozen installs it without re-resolving. A report that includes its uv.lock, or the commit from direct_url.json, reproduces with --frozen.
• One lock across TP peers: the first host resolves main and records it in uv.lock; the peers copy that file and run uv sync --frozen, so every rank installs the same commit and can confirm it from direct_url.json.
• Recovery from a bad update: check out the previous uv.lock and run uv sync --frozen to restore the last working commit. Changing @main to @<commit> in pyproject.toml holds it there until main is fixed.

Agreed that CI proves resolution, build, install and imports only, and that kernel fixes still need a re-export. If a pinned default is preferred, I can keep the pin and put main tracking behind an explicit command instead.

[original-el8]

@brandonmmusic-max GitHub reports merge conflicts with main; local git merge-tree confirms them for PR head 3ce4caf3b337 against main at 91cfb3695f10. Conflicting files:

• docs/cli-design.md
• docs/model-preparation.md
• ignition/patches/__init__.py
• pyproject.toml
• tests/preparation/test_setup_preparation.py
• uv.lock

Please reconcile these files with main and rerun the checks affected by the resolution.

[brandonmmusic-max]

GitHub reports merge conflicts with main; local git merge-tree confirms them for PR head 3ce4caf3b337 against main at 91cfb3695f10. Conflicting files:

• docs/cli-design.md
• docs/model-preparation.md
• ignition/patches/__init__.py
• pyproject.toml
• tests/preparation/test_setup_preparation.py
• uv.lock

Please reconcile these files with main and rerun the checks affected by the resolution.

I’m finally past the Kiddo’s birthday party this weekend. I’ll get this done tonight

[brandonmmusic-max]

Merged main at d2b5c5f, which contains 91cfb36, in 36fd8d1. Main replaced ignition.patches with ignition.runtime_patches and pinned FlashInfer at e0f105b3; those two changes caused the six conflicts. The resolution keeps main's runtime patches and single-call setup script and keeps this PR's FlashInfer main dependency:

• pyproject.toml: the @main requirement with [tool.uv] upgrade-package, replacing main's e0f105b3 pin.
• uv.lock: main's lock with only the FlashInfer source changed, to ?rev=main#ea74ea61, the head of FlashInfer main. That commit contains e0f105b3's change as 07132dcab (Fix NaN sign ordering in b12x radix selection local-inference-lab/flashinfer#82, identical tree) plus three later commits: c08707c2b, 47946d885, and ea74ea610.
• ignition/patches/__init__.py: deleted, as on main. This PR changed only its --locked wording, and the ignition.runtime_patches messages name no uv command.
• tests/preparation/test_setup_preparation.py: main's single-call tests, with this PR's complete-command assertion and its --locked and --frozen exclusions.
• docs/cli-design.md and docs/model-preparation.md: main's runtime-patch text with this PR's FlashInfer text. The setup-script paragraph states that the script updates FlashInfer to the head of main and rewrites its uv.lock entry.

At ea74ea61, the FlashInfer main workflow ran out of disk on the hosted runner (run 42553335278 in brandonmmusic-max/flare-review, on 36fd8d1): the FlashInfer source tree is 1.01 GB, against 0.45 GB at 1f8ed8a3, and the lock and install steps each fetched it into a separate uv cache. a79c43c removes the runner's unused Android, .NET, GHC, and CodeQL toolchains first and points every uv command at one cache. Resolution, installation, and the assertions are unchanged.

Checks rerun:

• GitHub Actions on a79c43c in brandonmmusic-max/flare-review: the FlashInfer main job passed in 5m16s (run 42553853087). The lock matched git ls-remote main (ea74ea61), and FlashInfer 0.7.3 installed and imported at that commit. In Native serving (run 42553853071), rdma passed, and the Ubuntu and Windows jobs failed exactly the tests that fail on main at d2b5c5f (run 42544646868): legacy_templates_receive_image_blocks_and_return_original_sources on Ubuntu, and the same eight hub and load-package tests on Windows.
• Locally, without a GPU (uv 0.11.30, Python 3.12.3): uv lock --check exits 0. tests/preparation/test_setup_preparation.py and tests/preparation/test_packaging.py report 6 passed and 1 failed. The failure, test_server_cli_help_does_not_prepare_model, fails identically on main d2b5c5f: python -m ignition --help prints debug-serve on a line by itself, so the expected "\n debug-serve " text is absent.

No GPU run covers ea74ea61. The end-to-end check in the description used FlashInfer 1f8ed8a3, which FlashInfer main's history no longer contains, so that result does not cover the commit this lock records. The description is updated with these results and that limitation.

[original-el8]

@brandonmmusic-max GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head a79c43cc43b9 against main 97cfeb0001f1. Conflicting paths:

• AGENTS.md
• docs/model-preparation.md
• docs/roce-ring.md
• pyproject.toml
• tests/preparation/test_packaging.py
• uv.lock

Please reconcile the branch with main and rerun the checks affected by the resolution.