Llamacpp imatrix Quantizations of Qwen3-4B by Qwen
Using llama.cpp at commit 9a3bf2b for quantization.
Original model: https://huggingface.co/Qwen/Qwen3-4B
All quants made using the imatrix option with a bilingual (English + Chinese) and code-heavy calibration dataset, so low-bit quants keep more of their Chinese and coding ability than a English-only calibration would.
Run them in LM Studio, Ollama, or directly with llama.cpp and any llama.cpp based project.
Prompt format
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
Download a file (not the whole branch) from below:
| Filename | Quant type | File Size | Description |
|---|---|---|---|
| Qwen3-4B-Q8_0.gguf | Q8_0 | 4.28GB | Extremely high quality, generally unneeded but max available quant. |
| Qwen3-4B-Q6_K.gguf | Q6_K | 3.31GB | Very high quality, near perfect, recommended. |
| Qwen3-4B-Q5_K_M.gguf | Q5_K_M | 2.89GB | High quality, recommended. |
| Qwen3-4B-Q4_K_M.gguf | Q4_K_M | 2.50GB | Good quality, default size for most use cases, recommended. |
| Qwen3-4B-IQ4_XS.gguf | IQ4_XS | 2.27GB | Decent quality, smaller than Q4_K_S with similar performance, recommended. |
| Qwen3-4B-IQ3_M.gguf | IQ3_M | 1.96GB | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
| Qwen3-4B-Q2_K.gguf | Q2_K | 1.67GB | Very low quality but surprisingly usable. |
The .imatrix file used to produce these quants is included in this repo, so anyone can reproduce or extend the quant set with the exact same importance matrix.
Downloading using the hf CLI
First make sure you have the CLI installed:
pip install -U "huggingface_hub[cli]"
Then target the specific file you want (do not clone the whole repo, it is large):
hf download 6block/Qwen3-4B-GGUF --include "Qwen3-4B-Q4_K_M.gguf" --local-dir ./
Which file should I choose?
A great write up with charts showing various performances is provided by Artefact2 here.
The first thing to figure out is how big a model you can run. For that you need to know how much RAM and/or VRAM you have.
If you want the model running as FAST as possible, fit the whole thing in your GPU's VRAM: pick a quant with a file size 1-2GB smaller than your total VRAM.
If you want maximum quality, add your system RAM and your GPU's VRAM together, then pick a quant 1-2GB smaller than that total.
Next, decide between an 'I-quant' and a 'K-quant'.
If you don't want to think about it, grab a K-quant, in the format QX_K_X such as Q5_K_M.
If you want to dig deeper, see the llama.cpp feature matrix.
In short: if you are targeting below Q4 and running cuBLAS (Nvidia) or rocBLAS (AMD), look at the I-quants, in the format IQX_X such as IQ3_M. They are newer and offer better quality for their size. I-quants also work on CPU, but are slower than their K-quant equivalent, so it is a speed-vs-quality tradeoff.
Credits
Quantization pipeline built on llama.cpp by ggml-org.
Thanks to bartowski and kalomaze for establishing the imatrix calibration practice this pipeline follows.