K2-Horizon-32B Heretic GGUF
GGUF quants of a refusal-attenuated derivative of the IFM/K2-Horizon-32B model.
The pipeline is: Heretic refusal attenuation (trial 124: 3/100 refusals, KL divergence 0.056) → LoRA merge → GGUF conversion → quantization. This release includes all five precision levels.
Files
K2-Horizon-32B-heretic-BF16.gguf
K2-Horizon-32B-heretic-Q8_0.gguf
K2-Horizon-32B-heretic-Q6_K.gguf
K2-Horizon-32B-heretic-Q4_K_M.gguf
K2-Horizon-32B-heretic-Q2_K.gguf
| File | Size | Note |
|---|---|---|
| K2-Horizon-32B-heretic-BF16.gguf | ~69.6 GiB | Conversion source. Highest precision |
| K2-Horizon-32B-heretic-Q8_0.gguf | ~37.0 GiB | No imatrix. Needs 40+ GiB VRAM to fully offload |
| K2-Horizon-32B-heretic-Q6_K.gguf | ~28.5 GiB | No imatrix. Fits 2x16 GiB VRAM fully offloaded |
| K2-Horizon-32B-heretic-Q4_K_M.gguf | ~21.1 GiB | Recommended. imatrix-calibrated. Fits 2x16 GiB VRAM |
| K2-Horizon-32B-heretic-Q2_K.gguf | ~13.3 GiB | Smallest. imatrix-calibrated |
| Property | Value |
|---|---|
| Architecture | K2Horizon (custom) |
| Parameters | 34.8B dense |
| Format | GGUF |
| Context | 524,288 tokens max (reduce on small VRAM, see below) |
| License | Apache-2.0, inherited from the base model |
llama.cpp usage
A K2-compatible llama.cpp build is required (upstream does not support this
architecture). Use MBZUAI-IFM/llama.cpp
at branch model/K2Horizon.
Server example (2x16 GiB VRAM):
llama-server -m K2-Horizon-32B-heretic-Q4_K_M.gguf \
--split-mode tensor --tensor-split 1,1 \
--host 0.0.0.0 --port 8080 -ctk q4_0 -ctv q4_0 -c 8192
Note: -c 32768 needs about 17 GiB extra for KV cache and will OOM on
2x16 GiB cards. Keep -c 8192 with q4 KV cache on this class of GPU.
日本語
概要
IFM/K2-Horizon-32B をベースに、Heretic で無検閲化したモデルの GGUF です。 trial 124(拒否 3/100、KL 0.056)をマージしています。 BF16・Q8_0・Q6_K・Q4_K_M・Q2_K の5段階入りです。
使い方
llama.cpp は K2 対応フォークが必要です(本家は未対応)。
llama-server -m K2-Horizon-32B-heretic-Q4_K_M.gguf \
--split-mode tensor --tensor-split 1,1 \
--host 0.0.0.0 --port 8080 -ctk q4_0 -ctv q4_0 -c 8192
VRAM 16GB×2 での実用は Q4_K_M が目安です。
ctx を 32768 まで上げると KV で溢れるので、
長い ctx が要る場合は -c を抑えるか KV 量子化を使ってください。