Qwen3-4B — W2.79, QAD checkpoint
The quantization-aware-distillation checkpoint that on-policy distillation starts from, for the W2.79 arm. This is a starting point for training, not a finished model.
Code and recipe: MingZwhy/QAOPD.
This is a latent checkpoint — read before loading
The tensors here are bf16 and not quantized. Quantization-aware training keeps high-precision master weights and applies the quantizer inside the forward pass; what gets saved is the master copy. Loading this file directly gives you an unquantized model. It will not error, and it will score higher than the W2.79 model does.
Use it as STUDENT_MODEL for the OPD stage, which supplies the quantizer
configuration itself:
BITWIDTH=w2.79 STUDENT_MODEL=<this checkpoint> bash scripts/opd/run_math.sh
For a model you can load and evaluate, use the recovered checkpoint MingZwhy/Qwen3-4B-W2.79-QAOPD instead.
Quantization
| weights | mixed INT1.58 / INT4 in blocks of 256, 50% of blocks at INT4 → 2.79 bits effective |
| embedding, output head | INT4 |
| activations | INT8 |
| KV cache | 16-bit during OPD and evaluation |
License
Apache-2.0, inherited from Qwen3-4B.