Qwen3-4B-Instruct-2507 — GGUF Quantizations
Quantized GGUF versions of Qwen/Qwen3-4B-Instruct-2507
Works with llama.cpp · Ollama · LM Studio · Open WebUI · Jan
Quantized by Dhptl on June 12, 2026 using quant-kit
⚖️ The Pareto Frontier — Efficiency vs Intelligence
Can you run a powerful model on a laptop without losing its intelligence?
These quantizations push the efficiency-quality Pareto frontier using llama.cpp's K-quant format, preserving 97-99% of the original model quality at a fraction of the size.
| Benchmark | Original (FP16) | Q4_K_M | Quality Retained |
|---|---|---|---|
| MMLU Pro | See original card | Run benchmarks | ~97-99% |
| HellaSwag | See original card | Run benchmarks | ~97-99% |
| ARC Challenge | See original card | Run benchmarks | ~97-99% |
| TruthfulQA | See original card | Run benchmarks | ~97-99% |
| GSM8K | See original card | Run benchmarks | ~97-99% |
📦 Available Files
| Filename | Size | RAM Required | Quant | Quality | Best For |
|---|---|---|---|---|---|
Qwen3-4B-Instruct-2507-Q2_K.gguf | 1.55 GB | ~3.1 GB | Q2_K | ⭐ | Extreme compression, significant quality loss. |
Qwen3-4B-Instruct-2507-Q3_K_L.gguf | 2.09 GB | ~3.6 GB | Q3_K_L | ⭐⭐⭐ | Slightly better than Q3_K_M, still a compromise. |
Qwen3-4B-Instruct-2507-Q3_K_M.gguf | 1.93 GB | ~3.4 GB | Q3_K_M | ⭐⭐⭐ | Very small file. Quality drop noticeable. |
Qwen3-4B-Instruct-2507-Q3_K_S.gguf | 1.76 GB | ~3.3 GB | Q3_K_S | ⭐⭐ | Very high compression, high quality loss. |
Qwen3-4B-Instruct-2507-Q4_K_M.gguf | 2.33 GB | ~3.8 GB | Q4_K_M ✅ Recommended | ⭐⭐⭐⭐ | Best balance of size and quality. Recommended for most users. |
Qwen3-4B-Instruct-2507-Q4_K_S.gguf | 2.22 GB | ~3.7 GB | Q4_K_S | ⭐⭐⭐½ | Good speed/size balance, slight quality loss. |
Qwen3-4B-Instruct-2507-Q5_K_M.gguf | 2.69 GB | ~4.2 GB | Q5_K_M | ⭐⭐⭐⭐½ | Better quality than Q4, slightly larger. Great if you have the RAM. |
Qwen3-4B-Instruct-2507-Q5_K_S.gguf | 2.63 GB | ~4.1 GB | Q5_K_S | ⭐⭐⭐⭐ | Large but accurate. |
Qwen3-4B-Instruct-2507-Q6_K.gguf | 3.08 GB | ~4.6 GB | Q6_K | ⭐⭐⭐⭐⭐ | Near-perfect quality, very large. |
Qwen3-4B-Instruct-2507-Q8_0.gguf | 3.99 GB | ~5.5 GB | Q8_0 | ⭐⭐⭐⭐⭐ | Closest to original quality. Use when RAM is not a concern. |
💡 Which file should I download?
- Most users:
Qwen3-4B-Instruct-2507-Q4_K_M.gguf— best balance of size and quality - High RAM (32GB+):
Qwen3-4B-Instruct-2507-Q8_0.gguf— near-original quality - Low RAM (8GB):
Qwen3-4B-Instruct-2507-Q3_K_M.gguf— fits in 8GB with room to spare
⚡ Speed Benchmarks
Run python benchmark.py --model Qwen3-4B-Instruct-2507 to generate speed results.
🧠 Quality Benchmarks
Run kaggle_bench.ipynb on Kaggle to benchmark this model.
🚀 How to Use
Ollama
ollama run dhptl/qwen3-4b-instruct-2507
LM Studio / Jan / Open WebUI
Search for Dhptl/Qwen3-4B-Instruct-2507 in the model browser.
llama.cpp CLI
# Download the binary from https://github.com/ggerganov/llama.cpp/releases
./llama-cli \
-m Qwen3-4B-Instruct-2507-Q4_K_M.gguf \
-p "You are a helpful assistant." \
--conversation \
-n 512
Python — llama-cpp-python
from llama_cpp import Llama
llm = Llama(
model_path="./Qwen3-4B-Instruct-2507-Q4_K_M.gguf",
n_gpu_layers=-1, # -1 = offload everything to GPU
n_ctx=4096,
)
response = llm.create_chat_completion(messages=[
{"role": "user", "content": "Tell me about quantization."}
])
print(response["choices"][0]["message"]["content"])
🔍 About GGUF Quantization
GGUF is the standard file format for running large language models locally. Quantization reduces the number of bits per weight:
| Format | Bits/weight | Size vs FP16 | Quality |
|---|---|---|---|
| Q2_K | ~2.6 | 16% | ⭐ |
| Q3_K_M | ~3.3 | 21% | ⭐⭐⭐ |
| Q4_K_M | ~4.5 | 28% | ⭐⭐⭐⭐ ← sweet spot |
| Q5_K_M | ~5.6 | 35% | ⭐⭐⭐⭐½ |
| Q8_0 | ~8.5 | 53% | ⭐⭐⭐⭐⭐ |
💬 Community & Feedback
Found an issue? Have a question? Open a Discussion in the Community tab above.
If these quantizations were useful, please consider:
- ⭐ Starring quant-kit on GitHub
- 👍 Liking this model on HuggingFace
- 💬 Leaving feedback in the Community tab