Kimi-K3-GGUF

GGUF quantizations of moonshotai/Kimi-K3, mirrored at AtomicChat/Kimi-K3.

Kimi K3 is a ~2.9T parameter (104B active) mixture of experts model: 93 layers, 896 experts with 16 active per token, 7168 hidden size, 1M context, trained natively in MXFP4. The original only ships in MXFP4, so everything here comes from a direct MXFP4 to Q8_0 conversion, and the lower quants are requantized from that Q8_0.

Honesty first: this thing is enormous

Let's be upfront. This is a monster. The smallest quant we make, IQ1_S at ~1.56 bits, still weighs about 540 GB. That already blows past the 512 GB that even well specced machines top out at. To actually run any of these you need enough fast memory (RAM + VRAM combined) to hold the whole file. In practice that means a large RAM server or a serious multi GPU rig, not a laptop and not most workstations.

We know most people won't be able to run these locally, and that's okay. We made them for the few setups that can, for people who want to poke at the model, and as an archival base so the quants already exist the day the hardware or the tooling catches up. Rule of thumb: grab the biggest quant your memory can actually hold.

Does it even work?

Yes. After computing a quant, we run a full perplexity pass using llama-perplexity, and Q3_K_S came out at PPL 1.6066 +/- 0.02582. That is a healthy number. It means the quantized model is coherent and not broken (a broken quant gives perplexity in the tens or hundreds, or NaN).

Usage

We added text and vision support for kimi_k3 in our fork. It lives in this PR: AtomicBot-ai/atomic-llama-cpp-turboquant#58.

[!IMPORTANT] build with -DGGML_CUDA=ON if you have enough GPU for this to work!

git clone https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
cd atomic-llama-cpp-turboquant
git fetch origin pull/58/head:kimi-k3 && git checkout kimi-k3
cmake -B build -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-mtmd-cli

Ask a model anything:

./build/bin/llama-cli -m ./IQ1_S/Kimi-K3-IQ1_S-00001-of-00014.gguf --jinja \
    -p "What is the capital of France?"

Use Vision:

Download a projector, and a picture:

wget -c https://huggingface.co/AtomicChat/Kimi-K3-GGUF/resolve/main/mmproj-F16.gguf
wget -O /tmp/test.jpg https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/mtmd/test-1.jpeg

Then, run a model with projector using --mmproj flag and include image with --image:

./build/bin/llama-mtmd-cli -m ./IQ1_S/Kimi-K3-IQ1_S-00001-of-00014.gguf \
    --mmproj ./mmproj-F16.gguf --image /tmp/test.jpg \
    -p "Describe this image in detail." --jinja

Plain mainline llama.cpp will reject these files with an unknown architecture error until Kimi K3 support lands upstream (tracking: ggml-org/llama.cpp#26185).

Vision works now!

Kimi K3 is a native vision language model, and as of this update the vision path works end to end in our fork: MoonViT-V2 support in clip/mtmd plus the converted vision tower, published right here as mmproj-F16.gguf and mmproj-BF16.gguf at the repo root. One small mmproj pairs with any text quant above, so nothing has to be re-downloaded. Verified with Q3_K_S: the model reads real images and describes them accurately.

Quants, quality

[!NOTE]
Every number is measured against Q8_0 as a base

PPL(q) - perplexity of the quant itself. PPL / Q8_0 - the same number relative to the reference, so 1.00 is indistinguishable and 1.63 means 63 % more perplexed; KL div - how far the whole output distribution drifted, in nats. Top-1 match - how often the quant picks the same next token as Q8_0; read this one if you read only one. Δp RMS - how much its confidence moved, in percentage points.

These do not compare with numbers from other repositories: every publisher uses a different reference model and a different corpus.

Speed

Measured with llama-bench on a single CPU box, no GPU: AMD EPYC 9554P (64 cores, 128 threads, 3.10 GHz), 1.5 TB RAM, 110 threads, model fully resident in RAM (page cache).

Note the crossover. IQ1_S generates faster because token generation is memory-bandwidth bound and there is simply less weight to stream per token. It processes prompts slower because prompt processing is compute bound, and unpacking 1.5-bit codes costs more CPU per weight than 3-bit K-quants. Pick by workload: long prompts favor Q3_K_S, long generations favor IQ1_S.

About the imatrix

imatrix.dat was computed over 350 chunks of 512 tokens on a mixed corpus: bartowski's calibration_datav3, wikitext-2 train, source code, and Wikipedia. That breadth matters here: with 896 experts and only 16 firing per token, a narrow corpus leaves most experts undersampled, and their weights fall back to default quantization. Expert coverage on this run landed at 99.4 to 99.8 percent.

Source is the Q3_K_S quant, not the full Q8_0: at 2.9 TB the Q8 does not fit in our 1.5 TB of RAM, while Q3_K_S at 1.2 TB does and runs cleanly on CPU. Computing an imatrix from a quantized model is standard practice, and its effect on the final 1 to 2 bit outputs is second order next to the quantization itself.

Attribution and license

This is a derivative of Kimi K3 by Moonshot AI, redistributed under the Kimi K3 License. The original copyright notice and license are kept in LICENSE.

Kimi K3 License
Copyright (c) 2026 Moonshot AI

Quantized and published by AtomicChat.