🚀 Quick start
Tuned for a 24 GB card; what each flag does is below.
VLLM_ATTENTION_BACKEND=FLASHINFER \
VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=201326592 \
vllm serve pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 \
--max-model-len 106496 \
--kv-cache-dtype fp8_e4m3 \
--max-num-seqs 2 \
--gpu-memory-utilization 0.95 \
--max-num-batched-tokens 2048 \
--language-model-only \
--enable-prefix-caching \
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
--enable-auto-tool-choice --tool-call-parser qwen3_xml \
--reasoning-parser qwen3
--tool-call-parser qwen3_xmlis required for tool calling. Qwen3.8 emits XML tool calls (<tool_call><function=NAME>), not JSON. Without ittool_callsis always empty — which looks like a broken quant but is a serving flag.
Quality
Measured on vLLM against a bf16 reference served identically — the only valid control. (That same reference scores 0.563 here vs 0.494 on llama.cpp, so these do not compare across stacks.)
| bf16 (ref) | W4A16 | |
|---|---|---|
| Tool-selection accuracy · 174 held-out turns | 0.563 | 0.563 |
| Parameter accuracy | 0.345 | 0.350 |
| Schema-valid rate | 0.948 | 0.948 |
| Malformed tool calls | 0 | 0 |
| Median KLD vs bf16 · 6 distributions | — | 0.0021 – 0.0205 |
| Long-context retrieval · 29.7k tokens | — | 3/3 codes exact |
| Vision · colour + shape + position | — | 3/3 |
SWE-rebench dask__dask-11393 | — | resolved · 5 steps · 0 malformed |
Tool-calling is bf16-identical, turn for turn. KLD is lowest (0.0021) on agentic text — exactly where it was calibrated — and highest (0.0205) on general English.
Read these honestly. SWE-rebench is one instance, not a pass rate. Agentic deltas are
noise-bound: vLLM is nondeterministic at temperature=0, and replaying the ladder moved
levels by up to 6 turns (3.4pp) — read the endpoints, not the third decimal. Video,
multi-image and long-context vision are untested. The vision tower and MTP head are bf16,
not quantized (+1.65 GiB), which is why they work.
GPTQ W4A16 via llm-compressor 0.13.0 → compressed-tensors 0.18.0: int4, group 128, symmetric, static act-order. Sequential pipeline over all 64 layers (48 linear-attention + 16 full-attention), 496 modules quantized.
Kept bf16 on purpose: lm_head, embed_tokens, vision tower, MTP head. The
248,320-token vocab with untied embeddings makes the two vocab tensors 4.74 GiB, 26% of
the download — which is why a "4-bit" 27B lands at 5.2 bpw overall (trunk alone: 4.05)
and is bigger than the 14.5 GiB IQ4_XS GGUF. A quantized head over a 248k vocab is the
classic rare-token failure mode; the needle test is the check that this worked.
Calibration: 4,255,761 tokens / 3,436 windows at ctx 32,768 — 63% real agentic sessions (CLI logs + SWE trajectories), 6,786 tool calls across 76 schemas, plus reasoning turns, broad-instruct and red-team refusals. GPTQ drew 128 × 32,768-token sequences by deterministic whole-corpus stride.
Built with Quant-Tuner; logs mined with LogMiner.
Running it
The Quick start command is tuned for a 24 GB card (RTX 3090): 106,496 tokens of context at ~75 tok/s (45 without MTP), using 23.6 of 24.0 GiB. The download is 18.1 GiB and still fits because only 16 of the 64 layers keep a KV cache — the other 48 are linear attention.
| Flag | Why it matters |
|---|---|
--tool-call-parser qwen3_xml | Required for tool calls. Qwen3.8 emits XML, not JSON. Without it tool_calls is always empty — which looks like a broken quant but is a serving flag. |
--reasoning-parser qwen3 | Required for reasoning. Without it the closing </think> is dropped as a special token and reasoning arrives glued onto the answer. With it, reasoning goes to message.reasoning — that name, not reasoning_content. |
--kv-cache-dtype fp8_e4m3 | Buys the context: halves the cache to 32 KiB/token, worth ~+55K tokens. Needs FlashInfer — on a 3090 the other attention backends refuse 8-bit KV. |
--speculative-config … qwen3_5_mtp | The trained MTP draft head ships inside the checkpoint — no second file. ~1.7× decode (75 vs 45 tok/s). Use num_speculative_tokens: 3 on a 3090 (79 vs 74 tok/s); 2 was the optimum on Blackwell (128 tok/s, 1.71×, 71.2% acceptance). Re-tune per card. |
--language-model-only | Skips the vision tower, worth ~25K tokens of context. Drop it if you need image input. |
--max-num-seqs 2 | Each concurrent request costs ~148 MiB of recurrent state on top of KV cache. Raising it to 8 costs ~30K tokens of context. |
--gpu-memory-utilization 0.95 | Don't raise it. 0.98 measures 131K of context and then dies on the first request. |
Reasoning needs a 4096 budget. Reasoning and the answer share max_tokens, and
reasoning is spent first — there is no separate thinking budget. A hard problem spent 2,714
tokens thinking; at 1–2K it returns an empty answer with finish_reason: length, which
looks like a broken model but is only the budget running out. Set effort per request with
{"chat_template_kwargs": {"reasoning_effort": "low"}} (xhigh/high/medium/low), or
enable_thinking: false — which measured best for tool calling (0.563 vs 0.437 at xhigh).
Tool calls cost only ~22 reasoning tokens either way.
Chat template. Pass --chat-template chat_template_safe_v2.jinja (bundled). The stock
template raises on the OpenAI-standard reasoning_effort: "high", so a normal client gets
HTTP 400. The safe template fixes that plus three rendering bugs and is byte-identical on
382/382 real holdout prefixes, so adopting it cannot change quality.
The first start after changing any flag reports a smaller KV cache. vLLM compiles the
model on a cold cache and still holds that memory while measuring what is free, so you get
~2.2 GiB instead of ~4.3 GiB — often too little to start. The second launch is correct.
saved AOT compiled in the log means cold (restart it); Directly load AOT means the
numbers are real.
FlashInfer compiles kernels on first use, and a stock pip install vllm cannot: there
is no CUDA_HOME (the PyTorch wheels ship a CUDA toolkit but do not put it on PATH), the
bundled compiler and CUDA headers can be different versions, and the linker wants lib64,
libcudart.so and stubs/libcuda.so, which the wheels do not create. Install ninja, pin
nvidia-cuda-nvcc/crt/nvvm to match your PyTorch CUDA version, and symlink those three
paths.
Unrelated but expensive: pkill -f "vllm serve" kills your own shell, because the
shell's command line contains the pattern too. Use pkill -f "[v]llm serve".
License
Apache-2.0, inherited from Qwen/Qwen3.8-27B.