Qwen3.8-27B-QAT-Q2_0
This is a 2-bit (Q2_0) quantization-aware-trained (QAT) checkpoint of Qwen3.8-27B for llama.cpp at 8.76 GB.
Quantized by Samuel Yuan <sdkyuan@mit.edu>.
[!Note] This quantization was heavily focused on preserving code and reasoning performance. As such, multimodal and raw fact recall performance are slightly degraded.
Interestingly, being able to preserve reasoning and code performance at the expense of Wikitext recall performance during quantization via QAT seems to be in line with the "cognitive core" hypothesis.
A Q1_0 version and improved Q2_0 quant are in the works.
Unlike most community quants that use Post-Training Quantization (PTQ) (i.e. Unsloth), this is a quantization-aware-trained model.
Hopefully this quant makes Qwen3.8 27b-level coding and reasoning more accessible to those with <12-16 GB cards or Macs, since every little bit saved in model size means more room for KV cache and faster decode on memory bandwidth bound devices!
Experiments
We measure Top-1 agreement (how often the next-token argmax matches the FP16 teacher's) and mean KL divergence (nats/token, lower is better) on held-out chat-formatted slices and on wikitext-2.
Slices correspond to:
- Reasoning:
nvidia/Nemotron-Post-Training-Dataset-v1 (math),open-thoughts/OpenThoughts3-1.2M,nvidia/OpenMathReasoning (cot) - Code:
nvidia/OpenCodeReasoning,open-r1/codeforces-cots,nvidia/Nemotron-Post-Training-Dataset-v1 (code),bigcode/the-stack-smol - Tool calling:
Salesforce/xlam-function-calling-60k,togethercomputer/glaive-function-calling-v2-formatted,togethercomputer/CoderForge-Preview-32B-SWE-Bench-Verified-Evaluation-trajectories,nvidia/Nemotron-Post-Training-Dataset-v1 (tool_calling) - Wikitext-2:
Salesforce/wikitext(wikitext-2-raw-v1, test)
So this quantization exceeds the reasoning and code performance of SOTA PTQ quants over 1 GB larger, but does indeed suffer lower Wikitext accuracy. The comparatively lower tool calling performance may be due to our data blend, and we are looking to release a improved version with more focus on that as well for local agentic use cases.
Methods
We applied the Efficient QAT (https://arxiv.org/abs/2407.11062) recipe with some improvements to the E2E-QP step (KD from teacher instead of CE on dataset) and data blend.
We have included a mmproj file for vision. Again, note that multimodal capabilities may be slightly degraded.
Run it
./llama-cli \
-m qwen38-27b-qat-q2_0.gguf \
--jinja -cnv \
--reasoning-budget -1 \
-c 32768 -ngl 99 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
--flash-attn on \
--spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
--no-mmproj
or
./llama-server \
--model qwen38-27b-qat-q2_0.gguf \
--jinja \
--reasoning-budget -1 \
--ctx-size 32768 \
--host 127.0.0.1 \
--port 8033 \
-ngl 99 \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--flash-attn on \
--spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
--no-mmproj
Requires a llama.cpp build with Q2_0 support.
License
apache-2.0 inherited from the base model.
Feel free to reach out if you have any quesions about the model.