Observed behavior
With a8cb07c, a locally fine-tuned SmolLM2-360M Q4_0 GGUF failed inside FlareEngine.load with WebAssembly.RuntimeError: unreachable, before init_gpu() could run. Q8_0 loaded successfully. This is the browser CPU warmup/prefill path, not the existence of a Q4_0 WGSL decode kernel.
Reproduction and cause
A reproducible Q4_0 model is included in our public stix-agent-v0.1.0 release; the release lock pins the archive/files. The model file SHA-256 is b6bc4d842fbac3c9edc8c96c689f73ac834c17969adf03ebf75ac19d4d31114d.
Use that model with upstream flare-web bindings built from a8cb07c (resolve the nested-rustdoc JS syntax problem if using wasm-bindgen 0.2.117), then call:
await init();
const bytes = new Uint8Array(await (await fetch(localModelUrl)).arrayBuffer());
const engine = FlareEngine.load(bytes); // traps before GPU initialization
The release also contains our patched runtime; using that runtime will not reproduce the original failure.
The browser loader skips f32 copies of layer matrices, retaining raw quantized data. Its CPU warmup/prefill cannot consume those raw Q4_0 layer weights in the tested path, whereas Q8_0 works.
Compatibility workaround
Our WASM-only expansion converts each 18-byte Q4_0 block to a 34-byte Q8_0 block: preserve the f16 scale, emit the 16 low nibbles minus eight, then the 16 high nibbles minus eight as signed bytes. Decoded values are unchanged. Apply to initial loading and separate raw-weight attachment. Tests cover nibble order/scales and malformed lengths.
This is a compatibility workaround, not native Q4 CPU support: both CPU and GPU use the expanded representation and consume more memory. Direct CPU Q4_0 kernels or an explicit unsupported-format error may be better upstream fixes. Browser loading, warmup, prefill and async generation should be tested together. Related #161/#165 add a GPU kernel; #278 tests block dequantization; neither establishes this browser load path.
Downstream implementation and validation
We run a patched Flare build in STIX Meta Explorer. The publicly committed patch set applies to a8cb07cf54fa43c4e41664316498ed195fa7b018 (flare-web 0.2.21); its SHA-256 is b50f6a694c85e2a74c8a4b9a7690ffa103db01d85eb8014a47e4a373f360b3e4. This is a downstream patch set, not a standalone Flare fork or an upstream-ready PR. It contains several fixes; the relevant section is linked above.
Validation notes and build/training instructions describe the tested scope. Measurements were recorded during September 8–9 development; this report is being filed September 21 after checking that upstream main is still at the same SHA. This report does not claim coverage of every model, quantization, or GPU.
Observed behavior
With
a8cb07c, a locally fine-tuned SmolLM2-360M Q4_0 GGUF failed insideFlareEngine.loadwithWebAssembly.RuntimeError: unreachable, beforeinit_gpu()could run. Q8_0 loaded successfully. This is the browser CPU warmup/prefill path, not the existence of a Q4_0 WGSL decode kernel.Reproduction and cause
A reproducible Q4_0 model is included in our public stix-agent-v0.1.0 release; the release lock pins the archive/files. The model file SHA-256 is
b6bc4d842fbac3c9edc8c96c689f73ac834c17969adf03ebf75ac19d4d31114d.Use that model with upstream flare-web bindings built from
a8cb07c(resolve the nested-rustdoc JS syntax problem if using wasm-bindgen 0.2.117), then call:The release also contains our patched runtime; using that runtime will not reproduce the original failure.
The browser loader skips f32 copies of layer matrices, retaining raw quantized data. Its CPU warmup/prefill cannot consume those raw Q4_0 layer weights in the tested path, whereas Q8_0 works.
Compatibility workaround
Our WASM-only expansion converts each 18-byte Q4_0 block to a 34-byte Q8_0 block: preserve the f16 scale, emit the 16 low nibbles minus eight, then the 16 high nibbles minus eight as signed bytes. Decoded values are unchanged. Apply to initial loading and separate raw-weight attachment. Tests cover nibble order/scales and malformed lengths.
This is a compatibility workaround, not native Q4 CPU support: both CPU and GPU use the expanded representation and consume more memory. Direct CPU Q4_0 kernels or an explicit unsupported-format error may be better upstream fixes. Browser loading, warmup, prefill and async generation should be tested together. Related #161/#165 add a GPU kernel; #278 tests block dequantization; neither establishes this browser load path.
Downstream implementation and validation
We run a patched Flare build in STIX Meta Explorer. The publicly committed patch set applies to
a8cb07cf54fa43c4e41664316498ed195fa7b018(flare-web 0.2.21); its SHA-256 isb50f6a694c85e2a74c8a4b9a7690ffa103db01d85eb8014a47e4a373f360b3e4. This is a downstream patch set, not a standalone Flare fork or an upstream-ready PR. It contains several fixes; the relevant section is linked above.Validation notes and build/training instructions describe the tested scope. Measurements were recorded during September 8–9 development; this report is being filed September 21 after checking that upstream main is still at the same SHA. This report does not claim coverage of every model, quantization, or GPU.