K2-Horizon-MoVA-36B-A4B APEX Mini (GGUF)
This repository contains a compact GGUF quantization of IFM/K2-Horizon-MoVA-36B-A4B for local llama.cpp inference.
K2-Horizon-MoVA-36B-A4B is a sparse Mixture-of-Experts model with Mixture-of-Values attention (MoVA). It has 36B total parameters and activates approximately 4B parameters per token. The native context length is 524,288 tokens.
This is the vanilla upstream APEX Mini mixed-precision profile, regenerated for K2-Horizon with a K2-shaped importance matrix computed from a cleaned, high-quality multilingual calibration corpus. The profile prioritizes normal generation quality and CUDA compatibility over the earlier 11 GB experimental target: Q3_K routed experts in edge layers, IQ2_S routed experts in middle layers, higher-precision shared experts and attention, and standard F32/Q6_K tensors where appropriate.
File
| File | Size | SHA-256 |
|---|---|---|
K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf | 14,735,072,384 bytes (13.72 GiB) | A0B4FEBE2745AF83434B99672288C7B20E7BC47F950ACDB5DC250E7C74D233C4 |
Quick start — llama.cpp
K2-Horizon requires the MBZUAI-IFM model/K2Horizon llama.cpp branch, or another build that includes the k2-horizon architecture and tokenizer support. A vanilla llama.cpp release may not recognize this GGUF yet.
Interactive chat:
llama-cli \
-m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf \
-c 4096 \
-ngl 99 \
-cnv
For CPU-only inference, omit -ngl 99 (or set -ngl 0). The embedded chat template is used automatically.
The K2-Horizon GGUF carries IFM's own chat template; no external template file is required.
OpenAI-compatible local server:
llama-server \
-m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf \
-c 4096 \
-ngl 99 \
--host 127.0.0.1 \
--port 8080
Then send requests to http://127.0.0.1:8080/v1/chat/completions with any OpenAI-compatible client. Increase -c only when sufficient RAM/VRAM is available; the model's native maximum is 524,288 tokens.
Quantization profile
| Tensor type | Count |
|---|---|
| Q3_K | 310 |
| Q4_K | 147 |
| IQ2_S | 84 |
| Q5_K | 24 |
| F32 | 232 |
| Q6_K | 1 |
The profile was produced from the BF16 checkpoint with the upstream APEX Mini tensor map (k2-vanilla-apex-mini-48.txt) and a K2-Horizon calibration matrix. A short CUDA-server smoke test produced coherent instruction-following and factual responses on the special K2-Horizon llama.cpp build.
This file contains no TQ1_0 or TQ2_0 tensors. IQ2_S is intentional: it is the standard APEX Mini middle-expert type and has CUDA kernels in the required K2-Horizon build.
Calibration and imatrix process
The calibration source is the combined_all_medium set from the MIT-licensed
eaddario/imatrix-calibration dataset.
The Parquet source was exported to UTF-8 plain text and cleaned before use:
- Unicode NFKC and control-character normalization
- whitespace and blank-line normalization
- removal of samples shorter than 80 or longer than 16,000 characters
- removal of adult/erotic material, including multilingual English and Chinese terms
- removal of gambling, SEO, credential-request, repeated-token, and low-diversity spam
- exact duplicate removal
The resulting corpus contains 15,654 samples and is used as plain text. Its
local SHA-256 is
010725E8BFA24511B5FEE811E5AC066F6C7BDE41FB73E6FC1C7C24FED302B58D.
Using the special K2-Horizon llama.cpp build, the importance matrix was generated with 12 chunks, a 512-token context, automatic GPU fitting, no perplexity pass, and GGUF output:
llama-imatrix \
-m K2-Horizon-36B-BF16.gguf \
-f calibration-high-quality-combined-all-medium.txt \
-ngl auto \
--chunks 12 \
--no-ppl \
--parse-special \
-o k2-horizon-high-quality.imatrix.gguf
The matrix contains 609 tensor entries over 12 chunks. Its SHA-256 is
12AE83A745C36045C6929C0907F924F97CF75C4B4D8B867C73D11544E7D32982.
The final APEX Mini was then produced from BF16 with the upstream tensor map
and this matrix:
llama-quantize \
--tensor-type-file k2-vanilla-apex-mini-48.txt \
--imatrix k2-horizon-high-quality.imatrix.gguf \
K2-Horizon-36B-BF16.gguf \
K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf \
Q3_K_M
The resulting GGUF retains the model's embedded IFM chat template and requires the K2-Horizon llama.cpp architecture support described above.
License and attribution
This quantization is derived from IFM/K2-Horizon-MoVA-36B-A4B and follows the upstream Apache-2.0 license. Please read the upstream model card for the original model's capabilities, limitations, citation, and usage guidance.