Qwen3-Coder-30B-A3B-Instruct-AWQ
AWQ W4A16 (4-bit weights, group size 128) quantization of
Qwen/Qwen3-Coder-30B-A3B-Instruct,
produced with llm-compressor for serving on
a single 24 GB GPU with vLLM.
~15.5 GB — fits an RTX 4090 with room for a large KV cache.
Quantization
| Method | AWQ, W4A16, group size 128 |
| Format | compressed-tensors (pack-quantized) |
| Ignored | lm_head, MoE router (mlp.gate) |
| Calibration | 64 samples of HuggingFaceH4/ultrachat_200k, seq len 1024 |
| MoE | moe_calibrate_all_experts=True — every calibration token is routed through all 128 experts, so each expert is calibrated |
Qwen3-Coder-30B-A3B is a Mixture-of-Experts model (128 experts, 8 active per token) using standard GQA attention — so it serves on consumer Ada GPUs with the stock attention backends, no special kernels required.
Serving with vLLM
vllm serve dark-side-of-the-code/Qwen3-Coder-30B-A3B-Instruct-AWQ \
--max-model-len 65536 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.95 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20}'
Notes:
- At
--max-model-len 65536on a 24 GB card the KV pool holds ~118K tokens (~1.8x concurrency). The base model's native context is 262K; 64K is the practical cap at 24 GB. - Tool calling works via vLLM's first-class
qwen3_coderparser. - Non-thinking model — no reasoning parser needed.
- Pass
--override-generation-configif your client doesn't send sampling params, or vLLM will default totemperature=1.0, which is far too hot for coding.
Validation
Verified after quantization: correct code generation, working tool_calls through the
qwen3_coder parser, and coherent prose output.
Selected as the local agentic coding model in a head-to-head evaluation driven by the Pi coding agent, where it reliably executed a read/edit/test tool loop.