Introduction
We reduce Qwen3.5-4B inference latency on a single A10G by quantizing both the target model and its speculative-decoding drafter to INT4, while maintaining quality across all three gates (MMLU-Pro, IFEval, GPQA-Diamond). This repository reproduces our 3rd-place Efficient Qwen Competition (AdaptFM @ ICML 2026) entry end to end. It combines three techniques:
- Quantization-Aware Distillation (QAD): trains the INT4 target model to follow the original BF16 model's distribution, recovering the accuracy that post-training quantization (PTQ) loses.
- Two-Stage Training for Block-Diffusion Drafter: pretrains the DFlash drafter with the BF16 target, then finetunes it on the QAD-applied INT4 target so it aligns with the quantized target distribution better than direct QAD-target training does.
- Drafter Optimization: applies INT4 GPTQ and Sliding-Window Attention (SWA) to reduce drafter overhead while maintaining acceptance length.
Hardware: Final serving and evaluation run on a single NVIDIA A10G 24 GB GPU. Retraining the QAD target and drafter requires an 80 GB-class multi-GPU machine, with at least 2 GPUs and preferably 8 GPUs. The gated prompt dataset is a one-time ~42 GB download.
Quickstart (vLLM)
vllm serve nota-ai/Qwen3.5-4B-QAD-W4A16 \
--quantization compressed-tensors \
--trust-remote-code
- For speculative decoding with the companion W4A16 DFlash drafter, see the drafter card.
Citation
@article{kim2026quantize,
title = {Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B},
author = {Jaeyeon Kim and Jewon Lee and Bo-Kyeong Kim},
journal = {arXiv preprint arXiv:2607.04244},
year = {2026}
}
License & provenance
Apache-2.0, derived from Qwen/Qwen3.5-4B (Apache-2.0) with INT4 initialization from cyankiwi/Qwen3.5-4B-AWQ-4bit. Distillation data derives from NVIDIA's Nemotron-Post-Training-Dataset-v2 (gated; see its terms).