Qwen3.6-35B-A3B-MXFP4_MOE
Overview
This is an optimized GGUF of noctrex/Qwen3.6-35B-A3B-MXFP4_MOE-GGUF Q8_0 variant.
The only change from the source model is the re-quantization of the output.weight (lm_head) tensor from BF16 to a more compact format. Two variants are provided:
- Q8_0 output — reduces the tensor from ~970 MiB to ~515 MiB (~+9% TG throughput)
- MXFP4 output — reduces the tensor from ~970 MiB to ~257 MiB (~+15% TG throughput)
During token generation (batch size 1), this matrix is read in full for every single token, making it a significant bandwidth bottleneck. The smaller variants reduce memory bandwidth pressure proportionally, yielding higher token generation throughput with no measurable impact on prompt processing speed.
Benchmarks
Benchmarks done on llama.cpp with two AMD R9700 GPUs
Q8_0 output (18.42 GiB)
| Metric | Original (BF16 output) | Q8_0 output | Change |
|---|---|---|---|
| TG128 | 104.0 t/s | 113.6 t/s | +9.2% |
| TG2048 | 103.8 t/s | 113.5 t/s | +9.3% |
| TG8192 | 102.0 t/s | 111.0 t/s | +8.8% |
| PP128 | 1507 t/s | 1527 t/s | +1.3% |
| PP2048 | 3878 t/s | 3906 t/s | +0.7% |
| PP8192 | 5379 t/s | 5361 t/s | -0.3% |
MXFP4 output (18.17 GiB)
| Metric | Original (BF16 output) | MXFP4 output | Change |
|---|---|---|---|
| TG128 | 104.0 t/s | 119.3 t/s | +14.7% |
| TG2048 | 103.8 t/s | 119.1 t/s | +14.7% |
| TG8192 | 102.0 t/s | 116.6 t/s | +14.3% |
| PP128 | 1507 t/s | 1538 t/s | +2.1% |
| PP2048 | 3878 t/s | 3903 t/s | +0.6% |
| PP8192 | 5379 t/s | 5383 t/s | +0.1% |
Comparison across all variants
| Metric | BF16 output (18.87 GiB) | Q8_0 output (18.42 GiB) | MXFP4 output (18.17 GiB) |
|---|---|---|---|
| TG128 | 104.0 t/s | 113.6 t/s | 119.3 t/s |
| TG2048 | 103.8 t/s | 113.5 t/s | 119.1 t/s |
| TG8192 | 102.0 t/s | 111.0 t/s | 116.6 t/s |
| PP128 | 1507 t/s | 1527 t/s | 1538 t/s |
| PP2048 | 3878 t/s | 3906 t/s | 3903 t/s |
| PP8192 | 5379 t/s | 5361 t/s | 5383 t/s |
Model details
- Base model: Qwen/Qwen3.6-35B-A3B
- Source quantization: noctrex/Qwen3.6-35B-A3B-MXFP4_MOE-GGUF
- Architecture: Qwen3.5MoE (256 experts, top-8 routing, ~3B active params per token)
- File sizes: 18.42 GiB (Q8_0 output), 18.17 GiB (MXFP4 output) — source: 18.87 GiB
- GGUF quantization: MXFP4 for MoE expert weights, Q8_0 for attention and shared FFN weights, BF16 for all other tensors except
output.weightwhich is re-quantized (Q8_0 or MXFP4) - Total parameters: 34.66B (MoE, ~3B active)
Why re-quantize the output tensor?
The MoE expert weights in this model use MXFP4 (4-bit) quantization, which is very compact. However, the final language modeling head (output.weight) was left in BF16 — a 248320×2048 matrix consuming ~970 MiB. During token generation (batch size 1), this matrix is read in full for every single token, making it a significant bandwidth bottleneck.
Profiling with llama.cpp's PERF_LOGGER showed that the output.weight matrix-vector multiply consumed ~16% of total token generation time, entirely bottlenecked by memory bandwidth.
By re-quantizing this single tensor to a more compact format, the bandwidth savings translate directly into faster token generation.
Q8_0 output
Q8_0 (block size 32, int8 + f16 scale per block) shrinks the tensor to ~515 MiB — a ~47% reduction. This is a high-fidelity quantization format (1 byte per weight + a per-block f16 scale). For a projection matrix of this size (508M elements), the quantization noise introduced by Q8_0 is negligible. Informal text generation tests show no perceptible quality degradation.
MXFP4 output
MXFP4 (shared FP8 scale per 32-element block, 4-bit elements) shrinks the tensor to ~257 MiB — a ~73% reduction. This provides the smallest possible output tensor size, yielding the highest token generation throughput. The MXFP4 format matches the quantization already used for the MoE expert weights in this model. While this is a more aggressive quantization than Q8_0, informal text generation tests suggest only subtle quality differences in practice.
Acknowledgments Special thanks to noctrex for the source MXFP4 MoE GGUF quantization that this model is derived from.