fix(metax): fall back to Triton cache ops when vllm C extensions are missing - #319
Closed
tengqm wants to merge 1 commit into
Closed
fix(metax): fall back to Triton cache ops when vllm C extensions are missing#319tengqm wants to merge 1 commit into
tengqm wants to merge 1 commit into
Conversation
tengqm
force-pushed
the
fix/metax-empty-triton-cache-ops
branch
from
July 29, 2026 06:20
91dfc36 to
515106e
Compare
…missing When vllm is built with VLLM_TARGET_DEVICE=empty, torch.ops._C_cache_ops is not registered. The MetaX flash attention backend calls vllm._custom_ops.reshape_and_cache_flash which delegates to that C extension — causing AttributeError at runtime. vllm already ships a pure-Triton implementation (triton_reshape_and_cache_flash) for platforms without compiled extensions. This patch routes _custom_ops through the Triton fallback when _C_cache_ops is absent. The patch only fires when _C_cache_ops is missing — standard vllm wheels with compiled C extensions are unaffected. Note: _C activation ops (silu_and_mul etc.) do NOT need stubs here. The MetaX vendor backend (silu_and_mul_maca) already bypasses vllm's SiluAndMul/GeluAndMul classes and computes the activations directly with F.silu/F.gelu — no dependency on torch.ops._C. Tested on MetaX C550 (MACA 3.7.2.0, torch 2.8.0+metax): vllm 0.20.2 empty + vllm-plugin-FL + Qwen3-4B, eager + flash attn, inference succeeds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tengqm
force-pushed
the
fix/metax-empty-triton-cache-ops
branch
from
July 29, 2026 08:22
515106e to
d3ab6bf
Compare
tengqm
marked this pull request as draft
July 30, 2026 05:00
Contributor
Author
|
This PR is not ready. Some further investigation is needed. |
Contributor
Author
|
Superseded by #333. #319's guard ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When vllm is built with
VLLM_TARGET_DEVICE=empty,torch.ops._C_cache_opsis not registered. The MetaX flash attention backend callsvllm._custom_ops.reshape_and_cache_flashwhich delegates to that C extension — causingAttributeErrorat runtime:Root cause
vllm's
_custom_ops.pyhardcodestorch.ops._C_cache_ops.reshape_and_cache_flash(...)— a compiled CUDA C extension that doesn't exist in empty builds.Fix
vllm already ships a pure-Triton implementation (
triton_reshape_and_cache_flash) invllm.v1.attention.ops.triton_reshape_and_cache_flashfor platforms missing the compiled extension. This patch monkey-patches_custom_opsto route through the Triton fallback when_C_cache_opsis absent.The patch only fires when
hasattr(torch.ops, '_C_cache_ops')isFalse— standard vllm wheels with compiled C extensions are completely unaffected.Pattern
Follows the same pattern as the existing Triton kernel disablement in
patches/__init__.pyline 30-31.Testing
Verified on MetaX C550 (MACA 3.7.2.0, torch 2.8.0+metax):
VLLM_TARGET_DEVICE=emptyvllm servestarts and inference succeeds