Qwen3.6-27B Q4_K_M GGUF dequantized to BF16 for vLLM
Source GGUF: unsloth/Qwen3.6-27B-GGUF, file Qwen3.6-27B-Q4_K_M.gguf.
Base config/tokenizer: Qwen/Qwen3.6-27B. Weights come from the GGUF, not the HF base checkpoint.
Conversion: GGUF tensors were dequantized to fp32, Qwen GDN layout transforms were reversed, Qwen RMSNorm offset tensors were converted to HF format, and tensors were cast once to BF16 before writing sharded safetensors.
Validation on an H200 with vLLM 0.23.0:
- 851 GGUF tensors mapped to 851 safetensor keys.
- Transformers CUDA load and short generation passed. The base visual tower is intentionally absent; serve this as text-only with
--language-model-only. - Numeric checks passed with
max_abs_vs_bf16=0; quantized matrix error vs fp32 direct dequant was BF16-scale (max_rel_vs_fp32 ~= 0.00389). - vLLM
/v1/completionssmoke passed.
Recommended serve command:
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve exolabs/Qwen3.6-27B-Q4KM-dequant-bf16-vllm \
--served-model-name qwen36-27b-gguf-q4km-bf16 \
--dtype bfloat16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.85 \
--attention-backend triton_attn \
--gdn-prefill-backend triton \
--trust-remote-code \
--language-model-only