Kimi K3 2-bit Mixed-IQ2 GGUF
A downloadable, multimodal GGUF quantization of MoonshotAI/Kimi-K3. The routed experts use a mix of IQ2_XXS and IQ2_XS; the vision projector remains BF16.
| At a glance | |
|---|---|
| Base model | 2.8T-parameter MoE; 16 of 896 routed experts active per token |
| Text weights | 793 GiB in 10 GGUF shards |
| Vision projector | 862 MiB, BF16 |
| Context | Up to 1M tokens in the base architecture; usable context depends on available memory |
| Integrity | Per-file SHA-256 hashes in verification-receipt.json |
Quick start
Download the full repository and load the first shard with a recent llama.cpp build:
hf download islamassanov/Kimi-K3-Mixed-IQ2-GGUF \
--local-dir Kimi-K3-Mixed-IQ2-GGUF
llama-server \
-m Kimi-K3-Mixed-IQ2-GGUF/Kimi-K3-Mixed-IQ2-00001-of-00010.gguf \
--mmproj Kimi-K3-Mixed-IQ2-GGUF/mmproj-Kimi-K3-BF16.gguf \
-c 32768
llama.cpp finds the other nine shards automatically. Omit --mmproj for text-only use. Allow roughly 794 GiB for the files, plus memory for model weights, KV cache, and runtime buffers.
What I changed
Kimi K3's released expert weights are already MXFP4, so this is requantization of the released checkpoint, not quantization from BF16. I encoded 146,048 routed expert matrices as IQ2_XXS and upgraded 101,248 to IQ2_XS. The upgrades were chosen by measured reconstruction-error reduction within a fixed byte budget. Another 2,628 tensors stay at higher precision.
The key encoder change is a decoder-aligned IQ2_XXS scale fit. Stock IQ2_XXS fits scales using idealized magnitudes 1, 3, 5; its decoder reconstructs 1, 3.125, 5.375. I fit the scales to those actual decoder values. The GGUF format, decoder, block size, and inference cost are unchanged.
Against the stock IQ2_XXS encoder, this reduced source-weight reconstruction error by 5.57% at identical bytes on a frozen sample of 1,104 expert matrices spanning all 92 routed layers. Every sampled matrix improved. This is a weight-fidelity result, not a percentage gain in model accuracy.
Model evaluation
This GGUF scored 871 / 1,000 (87.1%) on OCRBench — 97.9% of Moonshot's published 89.0% base-model score.
Files and license
All 10 numbered text shards are required. File sizes and hashes, including the separate vision projector, are recorded in verification-receipt.json.
Use of this model is subject to the Kimi K3 license. Thanks to Moonshot AI for Kimi K3 and the llama.cpp/GGML maintainers for the GGUF and IQ2 tooling. Questions or corrections: islam@uni.minerva.edu.