Qwen3.6-35B-A3B UD-Q4_K_M GGUF dequantized to BF16 for vLLM
Source GGUF: unsloth/Qwen3.6-35B-A3B-GGUF, file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf.
Base config/tokenizer: Qwen/Qwen3.6-35B-A3B. Weights come from the GGUF, not the HF base checkpoint.
Conversion: GGUF tensors were dequantized to fp32, Qwen GDN layout transforms were reversed, Qwen RMSNorm offset tensors were converted to HF format, MoE gate/up expert tensors were fused into experts.gate_up_proj, and tensors were cast once to BF16 before writing sharded safetensors.
Validation on an H200 with vLLM 0.23.0:
- 733 GGUF tensors mapped to 693 safetensor keys because 40 expert gate/up tensor pairs are fused.
- Transformers CUDA load and short generation passed. The base visual tower is intentionally absent; serve this as text-only with
--language-model-only. - Numeric checks passed with
max_abs_vs_bf16=0, including fused MoE gate/up halves; quantized matrix error vs fp32 direct dequant was BF16-scale (max_rel_vs_fp32 ~= 0.00389). - vLLM
/v1/completionssmoke passed with Triton attention, Triton GDN prefill, and Triton MoE backend.
Recommended serve command:
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve exolabs/Qwen3.6-35B-A3B-UD-Q4KM-dequant-bf16-vllm \
--served-model-name qwen36-35b-a3b-gguf-udq4km-bf16 \
--dtype bfloat16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.85 \
--attention-backend triton_attn \
--gdn-prefill-backend triton \
--moe-backend triton \
--trust-remote-code \
--language-model-only