Gemma 4 12B — closest Q4_0 to BF16
30% lower KL than Google's official Q4_0 QAT and 2.2× closer to BF16 than Unsloth's imatrix Q4_0. Same size as Google. Standard Q4_0 runtime.
| Q4_0 | KL to BF16 ↓ | Top-1 agreement ↑ |
|---|---|---|
| QUASAR | 0.048 | 94.3% |
| Google QAT | 0.069 | 93.5% |
| Unsloth Q4_0 (imatrix) | 0.108 | 91.6% |
| Standard Q4_0 (llama-quantize, no imatrix) | 0.156 | 89.8% |
The W4A16 twin shows the same pattern under transformers: KL 0.034 to BF16 vs 0.044 for Google's W4A16 QAT (top-1 95.5% vs 94.8%).
ollama run hf.co/QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF
llama-cli -hf QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF
Better fidelity than the popular alternatives
QUASAR also gets lower KL than Unsloth's Q4_0 repack of Google's QAT checkpoint, and than LM Studio's Q4_K_M — a more expressive format:
| GGUF | KL to BF16 ↓ |
|---|---|
| QUASAR Q4_0 | 0.048 |
| Unsloth Google-QAT repack (Q4_0) | 0.053 |
| LM Studio Q4_K_M | 0.110 |
Same BF16 reference, prompts, llama.cpp build, and backend.
Full evaluation and reproduction details → EVAL.md
Downstream benchmarks
Higher point estimates on all four downstream benchmarks than Google's QAT model (measured on the W4A16 twin, which carries the same trained codes).
| Benchmark | QUASAR | Google QAT |
|---|---|---|
| MMLU | 70.4 | 67.8 |
| IFEval | 87.6 | 87.3 |
| ARC-Challenge | 51.3 | 50.1 |
| AGIEval | 24.2 | 22.6 |
| Average | 58.4 | 57.0 |
The MMLU gain is statistically robust: +2.6 points, 95% CI [+2.0, +3.2] (paired bootstrap over 14,042 questions). Full configs, BF16 results and CIs are in EVAL.md.
Native QAT GGUF
Native Q4_0 QAT, not a post-training re-quantization. The GGUF stores QUASAR's trained INT4 codes and scales directly; all 328 quantized tensors are verified against the W4A16 twin (native_q4_0_receipt.json, verifier in scripts/). No re-rounding and no custom runtime.
Multimodal support is included through the BF16 vision projector from google/gemma-4-12B-it.
Run it
- Ollama:
ollama run hf.co/QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF— the vision projector is picked up automatically. - LM Studio: search
QUASAR-QAT gemma-4-12Bin Discover. - llama.cpp, vision:
llama-mtmd-cli -hf QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF --image photo.jpg -p "Describe this image." --jinja - Thinking: on by default in the embedded Gemma 4 chat template; toggle with
--chat-template-kwargs '{"enable_thinking": false}'(llama-cli / llama-server). - Sampling: Gemma 4 defaults — temperature 1.0, top-k 64, top-p 0.95.
- Context: 262,144 tokens.
What's inside
| File | Size |
|---|---|
gemma-4-12B-it-QUASAR-Q4_0.gguf — Q4_0 linears (trained group-64 scales), Q6_K token embeddings, chat template embedded | 6.98 GB |
mmproj-gemma-4-12B-it-BF16.gguf — unmodified BF16 vision projector from google/gemma-4-12B-it | 175 MB |
native_q4_0_receipt.json, SHA256SUMS, EVAL.md, scripts/ (packer/verifier, eval harvest, paired bootstrap) |
QUASAR
QUASAR is loss-aware quantization-aware training. It improves the low-bit reconstruction during training, then exports to standard deployment formats with zero inference overhead.
This checkpoint was healed against the BF16 teacher for ~600M tokens: one epoch over ~377k prompts self-distilled from BF16 Gemma 4 12B-it with thinking on, fp32 master weights, INT4 group-64.
Related
- W4A16 compressed-tensors (vLLM) twin of this checkpoint: QUASAR-QAT/gemma-4-12B-it-QUASAR-W4A16-G64
- Gemma 4 E4B, same recipe: GGUF · W4A16 · collection · Gemma 4 E4B & 12B — QUASAR 4-bit QAT
- Base model: google/gemma-4-12B-it · Paper: arXiv:2608.13966 · Org: huggingface.co/QUASAR-QAT
License
Based on Gemma 4 by Google DeepMind. Upstream LICENSE/NOTICE apply.
Citation
@article{counathe2026quasar,
title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
journal={arXiv preprint arXiv:2608.13966},
year={2026}
}