K2-Horizon-MoVA-36B-A4B GGUF

Community GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B.

About K2-Horizon-MoVA-36B-A4B

K2-Horizon-MoVA-36B-A4B is a sparse Mixture-of-Experts model with Mixture-of-Values attention (MoVA). The upstream checkpoint has 36B total parameters and activates about 4B parameters per token. It advertises a native 524,288-token context window and is intended for reasoning, coding, agentic, and long-context workloads. See the official model card for the original model, serving recipes, prompt conventions, and upstream results.

Upstream K2-Horizon-MoVA-36B-A4B benchmark results; the image and scores are from the official model card.

Fidelity measurements

The table below compares every published GGUF file against the BF16 reference on a held-out WikiText pilot: eight chunks from wiki.test.raw and eight chunks from wiki.valid.raw, using a 4,096-token context and the same K2 llama.cpp runtime. Values are averaged across the two splits. Size (GB) uses decimal GB (file_size_bytes / 1,000,000,000). Lower Mean KLD, ΔPPL, and RMS Δp, and higher Top-1 agreement, indicate closer next-token behavior to BF16. The BF16 reference mean PPL was 6.815612 in this pilot. These are next-token fidelity measurements, not a direct percentage of capabilities retained and not a substitute for task-specific evaluation.

Bold rows mark practical sweet spots: Q5_K_M for extra fidelity headroom, Q4_K_M as the balanced default, and IQ3_M as the compact importance-aware option. Q8_0 remains the high-bit reference profile.

In this pilot, Q1_0 shows very large divergence from the BF16 reference; the extreme-compression profiles should therefore be treated as experimental.

The machine-readable results are available in reproducibility/quality-summary.tsv, with corpus hashes, evaluation settings, and runtime provenance in reproducibility/manifest.md.

Quick start

Use a K2-Horizon-capable llama.cpp build. The explicit partial offload and CPU MoE options in this example keep the model within the available accelerator memory when full offload is not possible; adjust the GPU-layer count for your hardware.

./llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q6_K.gguf \
  --chat-template-file reproducibility/chat_template_smoke_user.jinja \
  --jinja \
  --reasoning off \
  -p 'Answer briefly in English: What is GGUF, and why is it useful for running language models locally?' \
  -n 128 -c 4096 \
  --device CUDA0 \
  --gpu-layers 8 \
  --cpu-moe

The included template is the compatible single-turn template used by the release smoke test. The upstream full tool-aware Jinja template is not claimed as certified by this package. Long-context use may require substantially more memory than the short example above.

Reproducibility and validation

The GGUF files were quantized directly from the locked upstream BF16 input. Q8_0 was generated without an imatrix; the other ladder members used the model-specific combined importance matrix. Every published file passed the load/generate smoke test before upload. Runtime throughput data, when present, is supplementary and is recorded separately from the BF16 fidelity comparison above in reproducibility/runtime-summary.tsv.

The public package includes compact reproduction inputs and scripts. Raw conversion, imatrix, quantization, smoke-test, fidelity, and benchmark logs are kept locally under reports/ and are intentionally not uploaded.

Checksums for all published artifacts and public reproduction files are in SHA256SUMS.txt. The locked source revision, BF16 checksum, runtime commit, calibration inputs, and validation settings are in reproducibility/manifest.md.

License and attribution

The upstream model is released under the Apache License 2.0. Preserve upstream attribution and the included LICENSE when redistributing these derivative artifacts.

These are community GGUF quantizations, not an official IFM release or endorsement.