K2-Horizon-7B GGUF — imatrix quants
GGUF quantizations of IFM/K2-Horizon-7B, made directly from the publisher's BF16 GGUF rather than from the FP8 checkpoint.
Files
| File | Quant | Size | Guidance |
|---|---|---|---|
K2-Horizon-7B-Q8_0.gguf | Q8_0 | 8.92 GiB | Near-BF16 quality; largest and safest quant |
K2-Horizon-7B-Q6_K.gguf | Q6_K | 6.89 GiB | Strong quality/size trade-off |
K2-Horizon-7B-Q4_K_M.gguf | Q4_K_M | 5.21 GiB | Recommended compact quant |
K2-Horizon-7B-imatrix-WikiText2-100K.gguf | imatrix | 5.10 MiB | Reproducibility artifact |
Q6_K is the llama.cpp quant name corresponding to “Q6 K”; there is no Q6_K_M format.
SHA-256 checksums:
5a98a289aba5c8c99ef05c9261287f19e8f586fd679bab45b632eb86b47809bf K2-Horizon-7B-Q8_0.gguf
4f0449e4582118df22e401b1dd31dcece63d0e431695836e0c5812901e7c51ac K2-Horizon-7B-Q6_K.gguf
66371e17b1ee10b8ae36c6c1c9c8ebbb29ae80d006f8951e15dfc4bea082fb4a K2-Horizon-7B-Q4_K_M.gguf
4095ee9e7d91bc46e73955fc5139196c2fcae65099eef058b75a6f3b5fa25136 K2-Horizon-7B-imatrix-WikiText2-100K.gguf
Quantization quality
Q6_K and Q4_K_M were produced with a 252-tensor importance matrix collected over 200 × 512-token chunks (~100k tokens) from the WikiText-2 training split. The imatrix was collected with the Q8_0 model because the 16.76 GiB BF16 graph exceeded the local Metal working-set limit. Q8_0 is a close proxy to BF16 here (measured mean KLD 0.000467). Output and token embedding tensors intentionally use llama.cpp's normal mixed-quant fallback because imatrix collection does not process output.weight by default.
Small held-out WikiText-2 guardrail, measured against BF16 logits over two 256-token chunks with llama.cpp commit 35999d101cf2233fc54f09c3c8d599da7303ce02:
| Quant | PPL | Mean KLD vs BF16 | RMS Δp | Same top token |
|---|---|---|---|---|
| BF16 | 11.508836 ± 2.043381 | — | — | — |
| Q8_0 | 11.508721 ± 2.041464 | 0.000467 ± 0.000052 | 0.476% | 98.031% |
| Q6_K | 11.500250 ± 2.039763 | 0.004487 ± 0.000333 | 1.767% | 93.701% |
| Q4_K_M | 11.884325 ± 2.101731 | 0.025116 ± 0.002599 | 3.922% | 88.189% |
This is a deliberately small local regression test, not a general capability benchmark. Task-specific evaluation is still recommended.
Compatibility
K2 Horizon support is not yet in a released upstream llama.cpp build at the time of publication. Use the publisher's model/K2Horizon fork or a future upstream build that includes K2 Horizon support. These files were generated and validated with fork commit 35999d101cf2233fc54f09c3c8d599da7303ce02.
Example:
llama-cli \
-m K2-Horizon-7B-Q4_K_M.gguf \
-ngl 99 \
-c 32768 \
--temp 1.0 \
--top-p 0.95 \
-n 32768
The embedded chat template defaults to high reasoning effort, matching the publisher's recommendation. Context length is 524,288 tokens, but practical context depends on available memory and KV-cache settings.
Reproducibility
- Source repository revision:
7d69990f2d27e67a3f52aaf2e24d34c1dcfdc630 - Source file:
K2-Horizon-7B-BF16.gguf - Source SHA-256:
088c5d0814ef955d137fd1073ee3a68f6a53411fca44b75dc58a320640000444 - llama.cpp fork commit:
35999d101cf2233fc54f09c3c8d599da7303ce02 - Calibration corpus: WikiText-2 raw training split, first 200 chunks at context 512
- Evaluation corpus: WikiText-2 raw test split, first two chunks at context 256
Quantization commands:
llama-imatrix \
-m K2-Horizon-7B-Q8_0.gguf \
-f wiki.train.raw \
-o K2-Horizon-7B-imatrix-WikiText2-100K.gguf \
-ngl 99 -c 512 -b 512 -ub 512 \
--chunks 200 --no-ppl
llama-quantize --imatrix K2-Horizon-7B-imatrix-WikiText2-100K.gguf \
K2-Horizon-7B-BF16.gguf K2-Horizon-7B-Q6_K.gguf Q6_K
llama-quantize --imatrix K2-Horizon-7B-imatrix-WikiText2-100K.gguf \
K2-Horizon-7B-BF16.gguf K2-Horizon-7B-Q4_K_M.gguf Q4_K_M
License and credit
The original model is released by IFM under Apache-2.0. See the upstream model card for architecture, training, usage guidance, limitations, and citation details.