Qwen3.6-35B-A3B-INT4-W4A16
INT4 weight-only (W4A16) quantization of Qwen/Qwen3.6-35B-A3B — a hybrid Mixture-of-Experts model (256 experts, top-8, ~3B active) with GatedDeltaNet linear-attention, a vision tower and an MTP head.
Quantization
- AWQ activation-aware scale search + symmetric INT4 weights, group_size 32, MSE observer (llm-compressor).
- Quantized: the MoE expert FFNs only (the bulk of the weights).
- Kept bf16 (quality-sensitive):
self_attn, the router (mlp.gate),shared_expert, GatedDeltaNet (linear_attn),lm_head, embeddings, vision tower, MTP head. - Format:
compressed-tensors(pack-quantized). Full recipe:recipe.yaml.
Evaluation
Served as W4A16 (vLLM, thinking enabled, temperature 0.6):
| Benchmark | Score |
|---|---|
| GSM8K | 96.8% (242/250) |
| MMLU-Pro | 80.2% (401/500) |
Usage (vLLM)
# W4A16 — int4 expert weights, fp16 activations
vllm serve Avesed/Qwen3.6-35B-A3B-INT4-W4A16 \
--tensor-parallel-size 2 --trust-remote-code --reasoning-parser qwen3
On Ampere the same checkpoint can be served as W4A8 (int4 weights + int8 dynamic activations) with the vllm-ampere-optimized fork, which adds the INT8-activation MoE Marlin path:
vllm serve Avesed/Qwen3.6-35B-A3B-INT4-W4A16 \
--tensor-parallel-size 2 --marlin-input-dtype int8 --trust-remote-code --reasoning-parser qwen3
Quantized with vllm-ampere-optimized/quantize.