K2-Horizon-MoVA-36B-A4B — GGUF

GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B, produced with the MBZUAI-IFM fork of llama.cpp (model/K2Horizon branch) that adds native k2-horizon architecture support.

Architecture note: Upstream llama.cpp does not yet support the k2_horizon / K2HorizonForCausalLM architecture (PR in progress as of September 2026). You must use the MBZUAI-IFM fork linked above — or upstream once the PR merges — to load these GGUFs.

Model Overview

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family released by the Institute of Foundation Models (IFM) at MBZUAI on 3 September 2026. It is a Mixture-of-Experts (MoE) model augmented with a novel Mixture-of-Values (MoVA) attention mechanism, storing ~36 B parameters while activating only ~4 B per token.

What Is MoVA?

Mixture-of-Values (MoVA) applies the same sparse routing philosophy as MoE, but inside the attention mechanism: the value projections are replaced by a learned pool of value experts, and only a small subset is activated per token. This lets the model store a much richer set of representational "styles" in the value space without paying for all of them at inference time, complementing the feed-forward MoE sparsity.

Quantization Details

IQ quants (those beginning with IQ) were generated using an importance matrix computed on the Q2_K quant with 8 × 512-token chunks of WikiText-2 calibration data (Salesforce/wikitext).

All quants were produced from the BF16 GGUF published by IFM at IFM/K2-Horizon-MoVA-36B-A4B-GGUF. No safetensors conversion was performed; the IFM BF16 GGUF already includes tokenizer metadata and a llama.cpp-compatible chat template.

Hardware Requirements (Rough Guide)

These figures are weight-only estimates and exclude KV cache, runtime overhead, and context length. They are not minimum VRAM/RAM requirements.

Memory caveat: "4B active" means 4B parameters are computed per token, not that the model fits in 4B-worth of memory. All 36B weights must be resident in RAM/VRAM.

Running with llama.cpp

Prerequisite: build from the MBZUAI-IFM fork (model/K2Horizon branch) until upstream llama.cpp merges the PR.

git clone --depth 1 --branch model/K2Horizon \
    https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-k2horizon
cd llama.cpp-k2horizon
cmake -B build -DGGML_NATIVE=OFF
cmake --build build -j4

Basic inference:

./build/bin/llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf \
  -p "Explain the difference between MoE and MoVA architectures." \
  -n 1024 \
  --temp 1.0 \
  --top-p 0.95 \
  -c 8192

Serving (Original BF16 Model)

The original BF16 safetensors checkpoint can be served with vLLM or SGLang.

vLLM:

vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice

SGLang (validated on 2 × H200):

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tp 2 \
  --ep 2 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

Best Practices

1. Always use reasoning_effort="high" — all reported benchmark scores were obtained at high reasoning effort.
2. Recommended sampling: temperature=1.0, top_p=0.95, max_new_tokens ≥ 32 768.
3. Tool use / agent tasks: enable the k2_horizon tool-call parser when using the vLLM or SGLang API.
4. llama.cpp context: the model supports 512 K natively, but local inference is memory-constrained — use -c 8192 or -c 32768 unless you have sufficient RAM.

Benchmark Results

Results from the IFM model card. All scores in %. Bold = best in row. K2-Horizon-MoVA-36B-A4B uses high reasoning effort; other open models are in their respective reasoning modes.

Source & Links

• Original model: IFM/K2-Horizon-MoVA-36B-A4B
• Official BF16 GGUF: IFM/K2-Horizon-MoVA-36B-A4B-GGUF
• llama.cpp fork (K2-Horizon support): MBZUAI-IFM/llama.cpp @ model/K2Horizon
• IFM blog post: https://ifm.ai/blog/k2/
• vLLM recipe: https://recipes.vllm.ai/IFM/K2-Horizon-MoVA-36B-A4B
• SGLang cookbook: https://docs.sglang.io/cookbook/autoregressive/IFM/K2-Horizon

Quantization Pipeline

These GGUFs were generated with the following pipeline, mirroring the approach used for other large MoE checkpoints in this workspace:

1. Fork checkout & build — the MBZUAI-IFM model/K2Horizon branch is cloned to a dedicated directory and compiled with CMake (GGML_NATIVE=OFF for portability).
2. Source download — the IFM BF16 GGUF (~70 GiB) is downloaded from IFM/K2-Horizon-MoVA-36B-A4B-GGUF instead of converting from safetensors (a direct BF16 or Q8_0 conversion would exceed the ~109 GiB free on /tmp).
3. Imatrix — computed on the Q2_K quant (~10 GiB, fits in 15 GB RAM) with 8 calibration chunks of WikiText-2 to produce an importance matrix for the IQ quants.
4. Quantization — all standard and IQ quants are produced from the BF16 GGUF source; each file is deleted immediately after a verified upload to keep disk usage under BF16 (~70 GiB) + one quant at any given moment.
5. Resume support — a status.json file tracks which quants have been generated and uploaded so interrupted runs resume from where they left off.

License

The original model and these GGUF quantizations are released under the Apache 2.0 License. See LICENSE on the upstream repository for full terms.

Citation

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}