Qwen3-Coder-Next-GPTQ
GPTQ (error-compensated weight quantization) (W4A16) quantization of Qwen/Qwen3-Coder-Next, made with llm-compressor in the compressed-tensors format for efficient serving with vLLM and SGLang.
Chinese version: README_zh.md
Why W4A16 (and not the official FP8)
Qwen publishes only an FP8 checkpoint for this model. This repo is the 4-bit (W4A16) build: ~40 GiB on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left untouched (see the protected layers below), so generation tracks the base model.
Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the original on a sparse MoE where the wrong rounding on a router would cascade.
Model
| Base model | Qwen/Qwen3-Coder-Next |
| Architecture | qwen3_next MoE — 80B total, ~3B active, 48 layers, 512 experts, top-10 routing |
| Context | 256K |
| Method | GPTQ (error-compensated weight quantization), W4A16, group size 128 |
| Format | compressed-tensors (safetensors) |
| Size | ~40 GiB |
| Tool | llm-compressor |
| License | Apache-2.0 (inherited from base model) |
What is quantized, and what is protected
GPTQ quantizes weights column-by-column and compensates the residual error into the not-yet-quantized columns using second-order (Hessian) information. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision:
| Tensors | Kept | Reason |
|---|---|---|
*.mlp.gate (router) | bf16 | a compressed router sends tokens to the wrong experts |
lm_head | bf16 | the output projection; a global 4-bit type would squeeze it hardest |
| everything else (expert + attention linears) | W4A16 | the bulk of the weights, where 4-bit pays off |
Quality (perplexity)
Perplexity on wikitext-2-raw (test), n_ctx = 512, 12 chunks, measured with vLLM — the same recipe
across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling.
| Weights | PPL | vs bf16 |
|---|---|---|
| bf16 (master, reference) | 7.74 | — |
| this repo (GPTQ W4A16) | 9.34 | +20.5% |
Serving
vllm serve 6block/Qwen3-Coder-Next-GPTQ
The quantization scheme is declared in config.json and detected automatically — no extra flags. Works
with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it
comfortably; on 24 GB use tensor/pipeline parallel or offload.
Prompt format
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
Other quantizations
| Format | Repo | Note |
|---|---|---|
| AWQ (W4A16) | 6block/Qwen3-Coder-Next-AWQ | the other 4-bit build |
| FP8 (official) | Qwen/Qwen3-Coder-Next-FP8 | 8-bit, ~2× this size |
Quantized by 6block with llm-compressor. Weights are unmodified apart from quantization; all model behavior and license obligations follow the base model.