Qwen3.5-4B-Instruct — Hob Forge Edition (GGUF)
Qwen's excellent 4B, set up correctly for small GPUs and modest RAM — with everything measured, nothing projected. This is a Hob Forge edition: we didn't train this model (all credit to Qwen, Apache-2.0); we quantized it with a current toolchain, verified the chat template and tool-calling actually work, measured speed / memory / quality-loss for every file, and wrote the run guide we wished existed. If it's in a table below, we ran it on real hardware.
The headline you won't find on other GGUF pages: this is a hybrid-attention architecture (8 full-attention layers + linear-attention DeltaNet + a 1-layer MTP block) — its KV cache is ~5× smaller than a dense 4B. Measured: 8K context costs 256 MiB; 32K costs 1 GiB. Long context on tiny hardware is this model's superpower.
Which file? (measured on RTX 5070, -ngl 99, llama.cpp b368b24c)
| File | Size | Gen speed tg128 | Perplexity (wikitext-2, 120 chunks) | Note |
|---|---|---|---|---|
| Q4_K_M ⭐ | 2.58 GiB | 138 t/s | 10.030 ±0.157 | recommended default (imatrix) |
| Q5_K_M | 2.93 GiB | 131 t/s | 10.009 ±0.158 | quality step up |
| Q6_K | 3.31 GiB | 120 t/s | 9.887 ±0.155 | near-lossless |
| Q8_0 | 4.28 GiB | 102 t/s | 9.832 ±0.154 | reference quality |
| IQ4_XS | 2.39 GiB | 45 t/s ⚠ | 10.094 ±0.159 | smallest, but i-quant dequant is ~3× slower on RTX cards — only pick this if the last 200MB matters more than speed |
| F16 | 8.42 GiB | — | 9.871 ±0.155 | conversion source |
F16 baseline shows the whole ladder loses almost nothing: Q4_K_M sits +0.16 PPL from F16,
inside ~1σ. imatrix (for Q4_K_M and IQ4_XS) computed from a 2MB wikitext-2-train slice,
250 chunks — the calibration file ships in this repo (calib/).
Memory budget (measured, not projected)
Architecture-fixed overheads (identical for every quant): KV cache + 50 MiB recurrent-state
- ~70–96 MiB compute buffer.
| Context | KV cache | Total @ Q4_K_M | Total @ Q8_0 | Fits 4GB? | Fits 8GB? |
|---|---|---|---|---|---|
| 4K | 128 MiB | ~2.8 GiB | ~4.5 GiB | ✅ Q4 | ✅ |
| 8K | 256 MiB | ~2.9 GiB | ~4.7 GiB | ✅ Q4 | ✅ |
| 16K | 512 MiB | ~3.2 GiB | ~4.9 GiB | ✅ Q4 | ✅ |
| 32K | 1 GiB | ~3.7 GiB | ~5.4 GiB | ⚠ tight | ✅ |
Yes — 32K context, whole model, under 4GB at Q4_K_M. Every number from llama.cpp's
own allocator logs on our hardware.
Run it
# ollama
ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
# llama.cpp (recent build required — this arch needs 2026 support)
llama-cli -m Qwen3.5-4B-Instruct-Q4_K_M.gguf -st -p "your prompt" -ngl 99 -c 8192
Sampling (Qwen's recommendations, verified here): thinking mode temp 0.6, top_p 0.95, top_k 20; non-thinking temp 0.7, top_p 0.8, top_k 20. This model thinks by default —
it emits reasoning before answering. Turn it off: think:false (top-level, ollama API) /
enable_thinking=False (transformers) / strip <think> blocks client-side for llama.cpp.
Full walkthrough — zero-to-first-tool-call on an 8GB-class GPU and a standard desktop,
with the troubleshooting we earned building this — in RUNNING.md.
Provenance & method
- Base:
Qwen/Qwen3.5-4B(Apache-2.0) — untouched weights, full multimodal snapshot converted text-only. - Toolchain: llama.cpp convert (build 2026-08-15) + quantize (build 2026-08-12). Chat template verified by rendering tests (system/no-system/tools × both); tool-calling exercised with a real call.
- Evals here measure our files (quantization quality), not the model's intelligence — for capability benchmarks see Qwen's card. PPL runs used identical chunks across all quants.
- No training data involved; nothing to decontaminate. No abliteration anywhere in lineage.
Limits
A 4B is a 4B: strong for its size at chat, coding assistance, and tool use; not a frontier model. IQ4_XS speed caveat above. Vision components of the base are not included (text-only GGUFs). MTP block included in F16 but speculative decoding needs runtime support.
Hob Forge — the small-GPU and modest-RAM champion. Measured budgets, honest tables, real support. If something in this card doesn't reproduce on your machine, open a discussion — we answer.