Conversation
UD 动态量化会在同一 block 内混用 GGML 类型(如 ffn_gate=IQ4_XS 配 ffn_up=Q4_K),加载期 gate+up / q+k+v 合并规则要求两输入 ggmlType 相同而静默跳过。主层 forward 支持分离投影不受影响, 但 HasMtpWeights 硬性要求融合的 mtp.layers.N.mlp.gateup_proj, MTP 因此静默失效。 RemapQwen35GGUFMtpTask 对 2-D 的 MTP 张量改走 GGUFWeightReplaceForceFP16(反量化到 FP16),合并机制即可产出 gateup_proj 与 mergeqkv;MTP 单层 FP16 约 850MB,显存增量可控。 另在 HasMtpWeights 顶部加 FASTLLM_DEBUG_MTP 环境变量门控的 mtp.* 权重清单打印,用于此类问题诊断。 实测 Qwen3.8-27B-UD-Q4_K_XL.gguf + 3090: [Qwen3.5 MTP] enabled: layers=1, drafts_per_step=3 pos_accept_rate=[81.25%, 59.38%, 34.38%],decode ~44 tok/s。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
UD 动态量化的 GGUF(如
Qwen3.8-27B-UD-Q4_K_XL.gguf)会在同一个 block 内混用 GGML 类型——实测该文件全部 65 层都是ffn_gate = IQ4_XS、ffn_up = Q4_K。而加载期的权重合并规则(model.cpp中 gate+up → gateup、q+k+v → mergeqkv)要求各输入ggmlType一致,类型混合时canMerge = false静默跳过。主层 forward 支持分离投影,模型照常运行;但
HasMtpWeights()硬性要求融合的mtp.layers.N.mlp.gateup_proj.weight存在,于是--mtp静默失效:无报错、无[Qwen3.5 MTP]日志,任何混类型 UD 量化 GGUF 都无法使用 MTP。修复
RemapQwen35GGUFMtpTask(src/model.cpp)对 2-D 的 MTP 张量改走GGUFWeightReplaceForceFP16(复用现成的反量化通道),MTP 各矩阵类型统一为 FP16 后,合并机制即可正常产出gateup_proj与mergeqkv。MTP 栈通常只有 1 层,FP16 约 850MB,显存增量可控。另在
HasMtpWeights()顶部加了FASTLLM_DEBUG_MTP环境变量门控的mtp.*权重清单打印,方便此类问题诊断(默认关闭,无开销)。验证
Qwen3.8-27B-UD-Q4_K_XL.gguf + RTX 3090 24GB(
--device cuda --mtp 3):FASTLLM_DEBUG_MTP=1显示 15 个 mtp 权重,gate_proj/up_proj分离存在、无gateup_proj,HasMtpWeightsfalse,无任何 MTP 日志decode 约 3.1 token/次验证,实测 44 → 50-58 tok/s;17×23=391、代码/翻译等生成质量正常。
测试计划
--mtp 3:MTP enabled、接受率、生成质量--mtp 0)