Repository navigation
Conversation
…results RecordedPreparationSession's tuning report sums result.restored_candidates and result.rejected_candidates, but the flashinfer PreparationResult those results come from provides neither (its __init__ signature: plans, coverage, benchmark_calls, benchmark_closers, cache_hits, benchmarked_candidates, elapsed_seconds, compilation, parent_cute_compilations, parent_triton_compilations, overlapped_benchmark_samples, program_counts). Every multi-rank preparation crashes with AttributeError: 'PreparationResult' object has no attribute 'restored_candidates'. Did you mean 'benchmarked_candidates'? the moment the report is built - observed live on a 4x DGX Spark TP4 ring export, ~70 min into b12x autotune, killing the whole rank group. The fields are report-only; read them with getattr and default 0 so a flashinfer that starts publishing them is picked up unchanged, and one that does not (current) reports 0 instead of aborting the preparation. Validated: 4-rank export on 4x DGX Spark (GB10) proceeds past the report build that previously crashed the group.
original-el8
added
area:preparation
Collaborator
|
The supported FlashInfer dependency now provides both fields this PR makes optional. Flare main pins 1f8ed8a3c46e27d03cff54e08f23e9bc50476d57, whose PreparationResult implementation defines Marked |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a live crash in every multi-rank preparation.
RecordedPreparationSession's tuning report (ignition/preparation/tuning.py:360-361) sumsresult.restored_candidatesandresult.rejected_candidates, but thePreparationResultthose results come from (flashinfer'sb12x.preparation.types) provides neither field — its__init__signature isplans, coverage, benchmark_calls, benchmark_closers, cache_hits, benchmarked_candidates, elapsed_seconds, compilation, parent_cute_compilations, parent_triton_compilations, overlapped_benchmark_samples, program_counts. Result on a real 4-rank export:Observed live on a 4x DGX Spark (GB10) TP4 ring export, ~70 minutes into b12x autotune (91/91 comm.roce kernels ready) — the report build killed ranks 1 and 3 and took down the whole rank group.
Fix: read both with
getattr(result, ..., 0). The fields are report-only, so a future flashinfer that starts publishing them is picked up unchanged; the current one reports 0 instead of aborting the entire preparation.The introduced-by commit appears to be 8e4a1f6 (
Preset DeepSeek V4.1 packed GPU speculation, which added the report block).