Qwen3.6-35B-A3B MXFP4_MoE No-MTP GGUF
Base Model
Qwen3.6-35B-A3B by Alibaba Qwen Team — a 35B-parameter Mixture-of-Experts (MoE) model with only 3B active parameters per token. Apache 2.0 licensed. Supports 131K+ context, tool use, and multilingual tasks across 100+ languages.
Why No MTP?
Qwen3.6-35B-A3B ships with Multi-Token Prediction (MTP) heads for speculative decoding. On MoE models, MTP heads waste ~1-2 GB of VRAM with no measurable speed benefit — the bottleneck is expert routing through system RAM (when using --cpu-moe), not draft token verification. This build strips the MTP heads with --no-mtp to reclaim that VRAM for expert weights and KV cache, resulting in better performance on 16GB cards.
Quantization
- Format: MXFP4_MoE — OCP microscaled FP4 (E2M1 values with E8M0 power-of-two scaling per 32 values) for routed expert weights; Q8_0 for all non-expert tensors (attention, embeddings, shared FFN, norms)
- BPW: 4.57 bits per weight
- Size: 19.8 GB
- Portable: MXFP4 works on any hardware with FP4 support (not Blackwell-only like NVFP4)
Tested Settings (llama.cpp build 1324+, RTX 5060 Ti 16GB)
llama-cli -m qwen3.6-35b-a3b-mxfp4-moe-nomtp.gguf \
-ngl 99 --cpu-moe \
--ctx-size 131072 \
--flash-attn on \
--cache-type-k q8_0 --cache-type-v q8_0 \
-t 8 --temp 0.7 --top-k 20 --top-p 0.95
Performance Results
| Metric | Result |
|---|---|
| Prompt processing | 23.7 t/s |
| Token generation | 30.3 t/s |
| Context | 128K |
| KV Cache | Q8_0 |
| Test prompt | "Tell me about superposition and quantum computing" |
| Output quality | Coherent, thorough, no hallucination, no repetition |
Usage
llama.cpp CLI (text generation)
llama-cli -m qwen3.6-35b-a3b-mxfp4-moe-nomtp.gguf -ngl 99 --cpu-moe -c 131072 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 -p "Your prompt here"
llama.cpp Server (API)
llama-server -m qwen3.6-35b-a3b-mxfp4-moe-nomtp.gguf -ngl 99 --cpu-moe -c 131072 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --host 0.0.0.0 --port 8080
Requirements
- llama.cpp build 700+ with CUDA support
- NVIDIA GPU with 16GB+ VRAM (tested on RTX 5060 Ti 16GB)
--cpu-moeflag required — routes MoE expert weights to system RAM while keeping attention on GPU--allow-requantizeused during quantization for mixed-precision output--flash-attn onrecommended for memory efficiency at long context
Notes
- MXFP4_MoE keeps expert weights at MXFP4 while non-expert tensors (attention, shared FFN, embeddings) stay at higher Q8_0 precision — better quality on always-active parameters
- Model uses
--cpu-moebecause total weights (~19.8 GB) exceed 16GB VRAM; experts stream from RAM - Slightly faster than NVFP4 variant (30.3 vs 29.1 t/s gen) due to better MXFP4 decode paths on Blackwell
- For NVFP4 variant (all weights NVFP4, Blackwell-only), see NVFP4 repo
License
Apache 2.0 (inherited from base model)