Qwen3.6-35B-A3B MXFP4_MOE GGUF Dequantized BF16 for vLLM
This private artifact was converted from unsloth/Qwen3.6-35B-A3B-GGUF file Qwen3.6-35B-A3B-MXFP4_MOE.gguf into standard Hugging Face safetensors for vLLM serving.
The source GGUF mixes MXFP4 MoE expert tensors with K-quant/Q8/F32 tensors. The converter streams GGUF tensors, dequantizes them to fp32, applies the Qwen3.6/GDN tensor remaps, performs the required RMSNorm offset handling, fuses MoE gate and up expert tensors into mlp.experts.gate_up_proj, then casts once to BF16 before writing sharded safetensors.
Validation
- GGUF dry-run mapping: 733 input tensors, 0 skipped, 0 unmapped.
- MXFP4 probe:
blk.0.ffn_gate_exps.weightdequantized successfully to fp32 with shape(256, 512, 2048). - Output index: 693 HF keys, 21 safetensor shards,
69,321,221,376bytes. - Transformers load: passed as
AutoModelForImageTextToTextwith expected missing visual tower weights only. - Numeric spot-checks: passed with
max_abs_vs_bf16=0for sampled direct, norm-offset, reshape/GDN, K-quant expert, and MXFP4 fused expert tensors. - Transformers generation smoke: passed.
- vLLM smoke: passed on vLLM 0.23.0 with
--language-model-only.
vLLM command used
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/model --served-model-name qwen36-35b-a3b-gguf-mxfp4_moe-bf16 --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.85 --attention-backend triton_attn --gdn-prefill-backend triton --moe-backend triton --trust-remote-code --language-model-only
The visual tower files are intentionally absent because the GGUF source contains language weights only; use vLLM text serving with --language-model-only.