Skip to content

Add Qwen3-VL tool calling support#5469

Merged
qgallouedec merged 23 commits intomainfrom
qwen3vl-tool-calling
Apr 10, 2026
Merged

Add Qwen3-VL tool calling support#5469
qgallouedec merged 23 commits intomainfrom
qwen3vl-tool-calling

Conversation

@qgallouedec
Copy link
Copy Markdown
Member

@qgallouedec qgallouedec commented Apr 7, 2026

  • Add response schema for Qwen3-VL (done by Claude)
  • Add qwen3_vl.jinja template for identity matching in add_response_schema
  • Add Qwen3-VL to TestAddResponseSchema and TestParseResponse test parametrizations
  • Add Qwen3-VL to supported models in agent training docs

Part of #5460

Warning

Requires/contains #5459


Note

Low Risk
Low risk: adds a new Qwen3-VL chat template and wires it into existing response-schema selection, plus test/doc updates; minimal impact outside Qwen3/Qwen3-VL parsing paths.

Overview
Adds Qwen3-VL tool-calling support by introducing a new qwen3_vl.jinja chat template (including multimodal image/video placeholders) and treating it as Qwen3-equivalent for add_response_schema so parse_response can extract tool calls.

Updates unit tests to cover Qwen3-VL parsing/schema behavior and refreshes Qwen3 test fixtures, and documents Qwen3-VL as a supported model for GRPO agent training.

Reviewed by Cursor Bugbot for commit 535544b. Bugbot is set up for automated code reviews on this repo. Configure here.

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


class TestIsChatTemplatePrefixPreserving:
def test_prefix_preserving_template(self):
tokenizer = AutoTokenizer.from_pretrained("trl-internal-testing/tiny-Qwen3MoeForSequenceClassification")
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same template, but it's more natural to use the CausalLM instead of the SequenceClassification

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't remember why we used Qwen3MoeForSequenceClassification in the first place

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 91e940e. Configure here.

@qgallouedec qgallouedec merged commit c73c2ec into main Apr 10, 2026
11 of 13 checks passed
@qgallouedec qgallouedec deleted the qwen3vl-tool-calling branch April 10, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants