Skip to content

fix(metax): fall back to Triton cache ops when vllm C extensions are missing - #319

Closed
tengqm wants to merge 1 commit into
mainfrom
fix/metax-empty-triton-cache-ops
Closed

fix(metax): fall back to Triton cache ops when vllm C extensions are missing#319
tengqm wants to merge 1 commit into
mainfrom
fix/metax-empty-triton-cache-ops

Conversation

@tengqm

@tengqm tengqm commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Problem

When vllm is built with VLLM_TARGET_DEVICE=empty, torch.ops._C_cache_ops is not registered. The MetaX flash attention backend calls vllm._custom_ops.reshape_and_cache_flash which delegates to that C extension — causing AttributeError at runtime:

AttributeError: '_OpNamespace' '_C_cache_ops' object has no attribute 'reshape_and_cache_flash'

Root cause

vllm's _custom_ops.py hardcodes torch.ops._C_cache_ops.reshape_and_cache_flash(...) — a compiled CUDA C extension that doesn't exist in empty builds.

Fix

vllm already ships a pure-Triton implementation (triton_reshape_and_cache_flash) in vllm.v1.attention.ops.triton_reshape_and_cache_flash for platforms missing the compiled extension. This patch monkey-patches _custom_ops to route through the Triton fallback when _C_cache_ops is absent.

The patch only fires when hasattr(torch.ops, '_C_cache_ops') is False — standard vllm wheels with compiled C extensions are completely unaffected.

Pattern

Follows the same pattern as the existing Triton kernel disablement in patches/__init__.py line 30-31.

Testing

Verified on MetaX C550 (MACA 3.7.2.0, torch 2.8.0+metax):

  • vllm 0.20.2 built with VLLM_TARGET_DEVICE=empty
  • vllm-plugin-FL main branch
  • Qwen3-4B eager mode, flash attention — vllm serve starts and inference succeeds
prompt_tokens=9, completion_tokens=32 ✅

@github-actions github-actions Bot added the core label Jul 28, 2026
@tengqm
tengqm force-pushed the fix/metax-empty-triton-cache-ops branch from 91dfc36 to 515106e Compare July 29, 2026 06:20
…missing

When vllm is built with VLLM_TARGET_DEVICE=empty, torch.ops._C_cache_ops
is not registered. The MetaX flash attention backend calls
vllm._custom_ops.reshape_and_cache_flash which delegates to that C
extension — causing AttributeError at runtime.

vllm already ships a pure-Triton implementation
(triton_reshape_and_cache_flash) for platforms without compiled
extensions. This patch routes _custom_ops through the Triton fallback
when _C_cache_ops is absent.

The patch only fires when _C_cache_ops is missing — standard vllm
wheels with compiled C extensions are unaffected.

Note: _C activation ops (silu_and_mul etc.) do NOT need stubs here.
The MetaX vendor backend (silu_and_mul_maca) already bypasses vllm's
SiluAndMul/GeluAndMul classes and computes the activations directly
with F.silu/F.gelu — no dependency on torch.ops._C.

Tested on MetaX C550 (MACA 3.7.2.0, torch 2.8.0+metax):
vllm 0.20.2 empty + vllm-plugin-FL + Qwen3-4B, eager + flash attn,
inference succeeds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tengqm
tengqm force-pushed the fix/metax-empty-triton-cache-ops branch from 515106e to d3ab6bf Compare July 29, 2026 08:22
@tengqm
tengqm marked this pull request as draft July 30, 2026 05:00
@tengqm

tengqm commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

This PR is not ready. Some further investigation is needed.

@tengqm

tengqm commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #333. #319's guard (hasattr(torch.ops, "_C_cache_ops")) is always True since the namespace is lazily auto-created, so the patch never fired; it also routed to vLLM's Triton op, which MetaX C550 disables. #333 registers reshape_and_cache_flash as a flag_gems dispatch op instead. Closing.

@tengqm tengqm closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant