Skip to content

Fix Qwen3 RotaryEmbedding fusion emitting invalid position_ids shape#29632

Open
Sammy-Dabbas wants to merge 1 commit into
microsoft:mainfrom
Sammy-Dabbas:fix-qwen3-rope-position-ids-format0
Open

Fix Qwen3 RotaryEmbedding fusion emitting invalid position_ids shape#29632
Sammy-Dabbas wants to merge 1 commit into
microsoft:mainfrom
Sammy-Dabbas:fix-qwen3-rope-position-ids-format0

Conversation

@Sammy-Dabbas

Copy link
Copy Markdown
Contributor

The on-the-fly RoPE fusion added for Qwen3 in #27590 wires the traced position_ids into the com.microsoft.RotaryEmbedding node without adapting its shape. A real Qwen3 export has no position_ids input, so HuggingFace builds one with a hardcoded batch axis of 1, shape 1 by sequence_length. The kernel only accepts a scalar or single-element position_ids, or an exact batch_size by sequence_length tensor, so any request with batch_size greater than 1 fails with "Input 'position_ids' dimension 0 should be of size batch_size, got 1". The existing tests missed it because the Qwen3 test generator declared position_ids as an already correct batch_size by sequence_length input.

In on-the-fly RoPE the position_ids is just the contiguous arange, and the kernel's scalar path derives every position from the first element, which reproduces that arange exactly and is batch-independent. This flattens the traced tensor and slices its first element to an equivalent single-offset input, so the fused node is correct at any batch size. A standalone check confirms the reduced output is bit-identical to the intended full arange, and a regression test verifies the fusion now emits it.

Fixes #29618.

The on-the-fly RoPE fusion added for Qwen3 in microsoft#27590 wires the traced position_ids into com.microsoft.RotaryEmbedding without adapting its shape. A real Qwen3 export has no position_ids input, so HuggingFace builds one with a hardcoded batch axis of 1, shape 1 by sequence_length, which the kernel rejects for batch_size greater than 1. Reduce it to an equivalent single-offset tensor. Adds a regression test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen3 RotaryEmbedding fusion produces invalid position_ids shape when batch_size > 1

1 participant