K2-Horizon-MoVA-36B-A4B-GGUF

This repository contains GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B, a sparse Mixture-of-Experts model utilizing Mixture-of-Values Attention (MoVA). It features 36B total parameters with only ~4B activated per token, enabling near-small-model inference speeds with 36B-scale intelligence.

[!IMPORTANT] Architecture Support Required: As of early 2026, k2-horizon is a newly introduced architecture not yet merged into upstream mainline llama.cpp. To run or quantize these GGUFs, you must build from the official MBZUAI-IFM fork (model/K2Horizon branch). Running on vanilla llama.cpp will result in unknown model architecture: 'k2-horizon'.

Quantization Overview & Hardware Recommendations

Installation & Setup

1. Build llama.cpp with K2-Horizon Support

# Clone the official architecture fork
git clone -b model/K2Horizon [https://github.com/MBZUAI-IFM/llama.cpp.git](https://github.com/MBZUAI-IFM/llama.cpp.git)
cd llama.cpp

# Build with hardware acceleration (enable CUDA if running on NVIDIA GPUs)
cmake -B build -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release # Add -DGGML_CUDA=ON for NVIDIA GPU
cmake --build build -j$(nproc) --target llama-cli llama-server

Quickstart Guide

Running via llama-cli

./llama.cpp/build/bin/llama-cli \
  -m ./K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  -p "<|ifm|im_start|>user\nWrite an efficient Python script for asynchronous data fetching.<|ifm|im_end|>\n<|ifm|im_start|>assistant\n"

Prompt Format & Reasoning Traces

K2-Horizon uses ChatML-style delimiters with native thinking tags:

<|ifm|im_start|>system
You are a helpful assistant.<|ifm|im_end|>
<|ifm|im_start|>user
Your query goes here.<|ifm|im_end|>
<|ifm|im_start|>assistant
<ifm|think>
[Model generates intermediate reasoning steps here]
</ifm|think>
[Final output generated here]<|ifm|im_end|>