Qwen2.5-Coder-1.5B-Instruct (Q4_K_M GGUF)
A compact GGUF quantization of Qwen/Qwen2.5-Coder-1.5B-Instruct — a code-focused 1.5B instruction model.
This card covers the single file:
| Property | Value |
|---|---|
| File | Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf |
| Size | 940.37 MiB (986.05 MB) |
| Quantization | Q4_K_M (≈ 4.85 bits/weight) |
| Parameters | 1.54B |
| Architecture | Qwen2 |
| Context length | 32768 tokens |
| GGUF version | 3 (latest) |
Why this quantization
Q4_K_M is the sweet spot for this model on low-VRAM hardware: ~0.94 GB
of weights fit any 4 GB-class GPU (e.g. GTX 1650) with plenty of headroom for
the KV cache. It keeps the model fluent for code generation, debugging and
algorithm explanations while being a fraction of the F16 size (2.9 GB).
Unlike reasoning (R1-style) models, Qwen2.5-Coder answers directly — no mandatory chain-of-thought — so it is fast and responsive in interactive agents and never falls into a "thinking loop".
It was produced from the F16 source with llama.cpp, and the advertised
context was capped at 32768 tokens so the KV cache fits a 4 GB GPU when fully
offloaded (-ngl 999).
Usage
llama.cpp (CLI)
llama-cli \
-m Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf \
-p "Write a Python function to reverse a linked list." \
-n 256 \
-ngl 999 \
-c 32768
llama-server (OpenAI-compatible API)
llama-server \
-m Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf \
-ngl 999 \
-c 32768 \
--port 8080
# curl http://localhost:8080/v1/chat/completions ...
Docker Model Runner (docker model)
docker model package \
--gguf Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf \
--context-size 32768 \
qwen-coder
docker model run qwen-coder "Write a Python function to reverse a linked list."
Ollama
ollama create qwen-coder -f Modelfile # FROM ./Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf
ollama run qwen-coder
Quality expectations
Q4_K_M on a 1.5B coder model handles common programming tasks well: function
writing, debugging, algorithm explanations, and simple refactors. Larger or
highly nuanced codebases may exceed its capacity (1.5B is a small model).
Quality is noticeably higher than IQ-type quantizations of the same size; for
the same VRAM footprint prefer Q4_K_M over Q4_0/IQ3_XXS.
Credits & license
- Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct (Apache 2.0)
- Source GGUF: bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF (MIT)
- Quantized with llama.cpp
- This file is released under the Apache 2.0 license.