K2-Horizon-MoVA-36B-A4B GGUF quants
Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo
GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B (MoVA mixture-of-value attention, 100 experts / 8 active, 512K context).
Converted with the official k2-official llama.cpp branch (MBZUAI-IFM port, commit 35999d101).
Files
| File | Quant | Size |
|---|---|---|
k2horizon-q3_k_m.gguf | Q3_K_M | ~16.9 GB |
k2horizon-q4_k_s.gguf | Q4_K_S | ~18.7 GB |
k2horizon-q4_k_m.gguf | Q4_K_M | ~20.8 GB |
k2horizon-q5_k_m.gguf | Q5_K_M | ~25.4 GB |
k2horizon-q8_0.gguf | Q8_0 | ~39 GB |
Requirements
Use a llama.cpp build from the k2-official branch of MBZUAI-IFM/llama.cpp (or anything that merges that port). Mainline llama.cpp does NOT support the k2-horizon architecture.
Usage
llama-server -m k2horizon-q4_k_m.gguf -ngl 99 --n-cpu-moe 42 -c 8192 -b 4096 -ub 2048 -t 32
On consumer GPUs (12 GB VRAM class), keep attention on GPU and let expert weights spill to CPU/RAM via --n-cpu-moe N. Lower N = more expert layers on GPU = more VRAM used, faster decode. Tune N until your VRAM budget is respected. -ub 2048 matters for prefill throughput on the CPU-offload path.
Notes
- Plain K-quants from F16, no imatrix.
- All quants verified loading and generating on RTX 3060 12GB with the offload config above.