┌────────────────────────────────────────────────────────────────────┐
│ DeepHat-V1-7B · GGUF │
│ ─────────────────────────────────────────────────────────────── │
│ base DeepHat/DeepHat-V1-7B arch qwen2 (Qwen2.5-Coder-7B ft) │
│ domain cybersecurity · devops · code │
│ ladder Q3_K_M → Q8_0 + IQ4_XS imatrix ✓ (code+general) │
│ fidelity Q8 KLD 0.0019 · top-1 99.8% vs BF16 │
└────────────────────────────────────────────────────────────────────┘
DeepHat-V1-7B — GGUF quant ladder
A full, imatrix-calibrated GGUF ladder of DeepHat/DeepHat-V1-7B — a Qwen2.5-Coder-7B fine-tune focused on cybersecurity, devops and code. This is a faithful re-quantization (all credit for the model goes to the DeepHat team); what's added here is the imatrix ladder, an objective KLD fidelity table vs the BF16 reference, and an honest evaluation.
First-mover note: at pack time no GGUF of this model existed. Quants Q3_K_M → Q8_0 + IQ4_XS, each with an importance matrix so the low-bit tiers stay sharp.
✅ Recommended files
| Use case | File | Why |
|---|---|---|
| Best all-round (16 GB RAM) | deephat-v1-7b-Q4_K_M.gguf | safe default, 98.97% top-1 vs BF16 |
| Quality-first (24 GB+) | deephat-v1-7b-Q6_K.gguf | near-lossless, 99.55% top-1 |
| Smallest usable (8–12 GB) | deephat-v1-7b-Q3_K_M.gguf | still 98.0% top-1 |
| Compact + sharp | deephat-v1-7b-IQ4_XS.gguf | imatrix IQ, 4.2 GB |
| Archival / eval | deephat-v1-7b-Q8_0.gguf | effectively lossless (KLD 0.0019) |
📦 Files
| Quant | Bits (BPW) | Size |
|---|---|---|
Q3_K_M | 3-bit K | 3.81 GB |
IQ4_XS | 4-bit IQ | 4.22 GB |
Q4_K_M | 4-bit K | 4.68 GB |
Q5_K_M | 5-bit K | 5.44 GB |
Q6_K | 6-bit K | 6.25 GB |
Q8_0 | 8.50 | 8.10 GB |
📊 Metrics — fidelity vs BF16 reference
Every tier measured against the unquantized BF16 GGUF (KL-divergence + PPL ratio + Top-1 agreement, general-English eval, c=2048).
| Quant | PPL(Q)/PPL(bf16) | Max KLD | KLD p99 | Top-1 match |
|---|---|---|---|---|
| Q8_0 | 1.000004 | 0.0019 | 0.0005 | 99.82% |
| Q6_K | 1.000340 | 0.0340 | 0.0079 | 99.55% |
| Q5_K_M | 1.000881 | 0.0624 | 0.018 | 99.50% |
| Q4_K_M | 0.998457 | 0.2078 | 0.030 | 98.97% |
| IQ4_XS | 0.996500 | 0.4858 | 0.041 | 98.74% |
| Q3_K_M | 1.002918 | 0.9454 | 0.109 | 98.02% |
The quantization is clean top-to-bottom — Q8 is effectively lossless and even Q3_K_M keeps 98% argmax agreement with BF16.
🧮 Will it fit?
| RAM / VRAM | Comfortable pick |
|---|---|
| 8 GB | Q3_K_M (short context) |
| 12 GB | Q4_K_M |
| 16 GB | Q5_K_M / Q6_K |
| 24 GB+ | Q8_0, long context |
7B at Q4–Q6 runs fast on a laptop GPU (Apple Silicon / consumer NVIDIA) or CPU.
🚀 How to run it
llama.cpp
llama-cli -m deephat-v1-7b-Q4_K_M.gguf -ngl 99 -c 8192 \
-p "<|im_start|>user\nExplain how a SQL injection works and how to prevent it.<|im_end|>\n<|im_start|>assistant\n"
Ollama (a ready Modelfile with configurable context ships in this repo):
ollama create deephat -f deephat-8192ctx.Modelfile
ollama run deephat "Write a bash script that scans a subnet for open port 22."
Sampling: ChatML template, temperature 0.1–0.7, top_p 0.9, top_k 20, stop <|im_end|>.
⚠️ Good to know
- Chat template = upstream original (tool-calling preserved). These GGUFs embed DeepHat's original
Qwen tool-calling template (the XLAM/Qwen
toolsformat), so native function-calling works just like the source model. That template uses the Jinjatojsonfilter, which older ollama / llama.cpp builds cannot parse (they error "Unknown (built-in) filter 'tojson'"). If you hit that: update your runtime (recent ollama/llama.cpp handle it), or override the template with the plain-ChatMLscripts/chatml.jinjashipped here (--chat-templatein llama.cpp) — that variant loads everywhere but drops native tool-calling. - Domain model. DeepHat is tuned for security / devops / code, not general chit-chat or general software engineering. See the eval below.
🧪 Evaluation methodology
- Fidelity gate (passed): the KLD/PPL/Top-1 table above — every tier is faithful to BF16 (Q8 lossless).
- Agentic probe (reported honestly):
swe-mix— 30 SWE-bench Verified instances (6 continuity anchors + 24 discriminating, 10 repos; spec inswe_mix.json), run through Claude Code + agent-bridge.js → Ollama servingQ6_K, tool-call formatopenhands,temperature 0.1, in Docker. Result:resolved = 0 / 30. The model did emit tool calls on 22/30 instances but ran very short episodes (2–8 messages) — it doesn't sustain the long multi-step agentic loop a SWE-bench solve needs. Honest caveat: (1) this is a 7B — small models routinely score 0–2 on SWE-bench Verified in a local agentic harness; (2) SWE-bench is general-repo software engineering (Django/astropy/sympy), which is not DeepHat's domain (cybersecurity/devops). This number measures general agentic SWE ability, not the model's security/code strengths, and it is not comparable 1:1 to the official leaderboard. Date: 2026-07-12. Small/hard probe, relative signal only.
This repo is fidelity-gated (like a non-general-SWE model release): the quant quality is proven; the SWE number is published transparently rather than hidden.
🔁 Provenance & reproducibility
scripts/reproduce.sh— exact convert → template-fix → imatrix → quantize commands.scripts/chatml.jinja— the clean chat template that replaced the upstreamtojsonone.- imatrix corpus: ~793 KB, general text + Python/Rust/C code (386 chunks,
c=512). reports/artifact-sha256sums.txt— SHA-256 of every GGUF.metrics/quant-summary-with-kld.{json,csv}— the table above, machine-readable.swe_mix.json— the 30-instance probe spec.
🗒️ Changelog
- 2026-07-12 — initial release. Ladder Q3_K_M→Q8_0 + IQ4_XS, imatrix, KLD sweep, cleaned chat template, honest SWE-mix eval.
Credit
Model: DeepHat/DeepHat-V1-7B (Apache-2.0). This repo only provides GGUF conversions + fidelity measurements. All model capability is the DeepHat team's work.