Qwen3.5-9B Q4_K_M GGUF dequantized BF16 for vLLM
Private Exolabs checkpoint converted from
unsloth/Qwen3.5-9B-GGUF file Qwen3.5-9B-Q4_K_M.gguf.
The upstream Qwen/Qwen3.5-9B repository was used for config, tokenizer, chat
template, and generation config only. Weight tensors come from the Q4_K_M GGUF
and were dequantized in fp32, transformed back to HF/vLLM layout where
llama.cpp stores Qwen3.5 Gated DeltaNet tensors differently, then cast once to
BF16.
Validated on vLLM 0.23.0 with:
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve . \
--served-model-name qwen35-9b-gguf-q4km-bf16 \
--dtype bfloat16 \
--max-model-len 4096 \
--attention-backend triton_attn \
--gdn-prefill-backend triton \
--trust-remote-code \
--language-model-only
Smoke tests:
The capital of France is-> starts withParis.Q: What is 2+2? A:-> starts with4.