--allow-quant-from-quant produces completely corrupted output on qwen3_5 (hybrid-GDN) architecture models

[karthikgs-in]

Summary

Converting an already-quantized MLX 4-bit checkpoint with --allow-quant-from-quant produces completely corrupted output (mojibake, replacement characters, mixed-script garbage) at inference time, on every qwen3_5/qwen35moe (hybrid Gated-DeltaNet) model tested. A native bf16 source of the same architecture family converts and serves cleanly. This isolates the bug to the --allow-quant-from-quant path specifically, not the qwen3_5 architecture support in general.

Environment

• BaseRT v0.2.3 (macOS arm64 engine bundle)
• macOS 26.6.2, Apple M3 Max, 36 GB unified memory
• Installed via install.sh

Repro

Broken (quant-from-quant):

basert convert /path/to/mlx-community/Qwen3.8-27B-4bit \
  --target base-q4 --allow-quant-from-quant \
  --output qwen3.8-27b-baseq4.base

basert-serve qwen3.8-27b-baseq4.base --port 8453 --max-context 4096

curl http://127.0.0.1:8453/v1/chat/completions -d '{
  "model": "qwen3.8-27b-baseq4",
  "messages": [{"role": "user", "content": "Say hello in one short sentence."}],
  "max_tokens": 40, "temperature": 0.0
}'

Result: content empty; reasoning_content is mojibake/mixed-script garbage (e.g. "�� \"\\\",�.\", \\n��/#___%、__ �\\ure-fetch_ Mi-_{mui_\\�) & _\t_S�\naum"), finish_reason: "length".

Reproduced on a second, independent checkpoint (lmstudio-community/Qwen3.6-35B-A3B-MLX-4bit, MoE variant, arch: qwen35moe) with the identical symptom on the identical trivial prompt — not checkpoint-specific.

Working (native bf16 source, no flag needed):

# Download Qwen/Qwen3.5-4B (native bf16, same qwen3_5 architecture family)
basert convert /path/to/Qwen/Qwen3.5-4B \
  --target base-q4 \
  --output qwen3.5-4b-native-baseq4.base

basert-serve qwen3.5-4b-native-baseq4.base --port 8453 --max-context 4096

Same trivial prompt: coherent, fluent English output, correct reasoning trace. No corruption at all. This isolates the bug cleanly to --allow-quant-from-quant, not the qwen3_5 architecture or the engine generally.

What I checked before filing

Read base-convert/crates/base-readers/src/mlx.rs end to end. It contains this comment, which describes the exact symptom observed:

"MLX scales/biases are F16 on older checkpoints (pre mlx-lm 0.20) and BF16 on newer ones (Gemma 4 4-bit, recent Qwen3 MoE). Reading BF16 bytes as F16 silently returns wildly wrong exponents → corrupted dequant + degenerate decode (<pad>, mojibake). Dispatch on the stored dtype."

I checked the actual safetensors header of mlx-community/Qwen3.8-27B-4bit directly:

layers.0.linear_attn.in_proj_qkv.weight: dtype=U32 shape=[10240, 640]
layers.0.linear_attn.in_proj_qkv.scales: dtype=BF16 shape=[10240, 80]
layers.0.linear_attn.in_proj_qkv.biases: dtype=BF16 shape=[10240, 80]

.scales/.biases are genuinely BF16 — exactly the case the comment names. I then read read_half() and the full bit-unpacking loop in dequant_packed() and cross-checked the shape math against these real values (bits=4, group_size=64 → packed_in=640 → in_features = 640*32/4 = 5120; groups_per_row = 80 = 5120/64 ✓). On manual review this looks technically correct — proper Bf16/F16 dispatch via scales_info.dtype, correct little-endian bit extraction, correct per-tensor override resolution.

This suggests the reader (dequant-to-f32) is not where the bug is, and it more likely lives in the re-quantization/writer stage (f32 → BaseQ4 packing) or in the engine's inference-time BaseQ4 unpacking for these specific GDN tensor shapes (in_proj_a/b/qkv/z, A_log, dt_bias, conv1d — none of which I could inspect further since the engine binary is closed-source).

Attempted follow-up diagnostic (not completed)

I attempted --target bf16 --allow-quant-from-quant on the same broken 27B source (skips re-quantization entirely — if that's coherent, the bug is conclusively downstream of the reader). Had to abort: free disk dropped from 55 GB to 27 GB in under a minute (a full bf16 dequant of a 27B model approaches its ~54 GB native footprint), too risky to let continue on that machine. This diagnostic would be a fast way to confirm reader-vs-writer if re-attempted on a smaller qwen3_5 checkpoint.

Suggested next steps for maintainers

1. Try --target bf16 --allow-quant-from-quant on a small qwen3_5-family MLX checkpoint (e.g. a hypothetical mlx-community/Qwen3.5-4B-4bit, if one exists) — if that also corrupts, the bug is in the reader after all (despite my read of it); if it's coherent, the bug is confirmed to be in the writer or engine.
2. If (1) points to the writer: check the BaseQ4 packing path specifically for the GDN-specific tensors (in_proj_a/b/qkv/z) — these have unusual shapes/splits not present in classic dense-attention Qwen3, and may hit an edge case the packer doesn't handle correctly when the source values already carry quantization-induced structure (vs. a smooth bf16 distribution).
3. If (1) points to the engine: the BaseQ4 kernel that unpacks these tensors at inference time may be mis-indexing something specific to the GDN layer's tensor layout — happy to help test a debug build if useful.

Both broken .base files and full conversion/serve logs are available if useful for further debugging.