Qwen3.6-35B-A3B Q8_0 GGUF Dequantized BF16 for vLLM
This private artifact was converted from unsloth/Qwen3.6-35B-A3B-GGUF file Qwen3.6-35B-A3B-Q8_0.gguf into standard Hugging Face safetensors for vLLM serving.
The converter streams GGUF tensors, dequantizes them to fp32, applies the Qwen3.6/GDN tensor remaps, performs the required RMSNorm offset handling, fuses MoE gate and up expert tensors into mlp.experts.gate_up_proj, then casts once to BF16 before writing sharded safetensors.
Validation
- GGUF dry-run mapping: 733 input tensors, 0 skipped, 0 unmapped.
- Output index: 693 HF keys, 21 safetensor shards,
69,321,221,376bytes. - Transformers load: passed as
AutoModelForImageTextToTextwith expected missing visual tower weights only. - Numeric spot-checks: passed with
max_abs_vs_bf16=0for sampled direct, norm-offset, reshape/GDN, and fused expert tensors. - Transformers generation smoke: passed.
- vLLM smoke: passed on vLLM 0.23.0 with
--language-model-only.
vLLM command used
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/model --served-model-name qwen36-35b-a3b-gguf-q8_0-bf16 --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.85 --attention-backend triton_attn --gdn-prefill-backend triton --moe-backend triton --trust-remote-code --language-model-only
The visual tower files are intentionally absent because the GGUF source contains language weights only; use vLLM text serving with --language-model-only.