Kimi-K3 MLX: 160 experts, 2-bit (181 GB)
An MLX build of moonshotai/Kimi-K3 (2.78 T total,
104 B active) pruned to 160 of 896 routed experts per layer and quantized to 2-bit at group
128, so it runs resident on a 192 GB Apple Silicon machine. In short, about half the memory of the
smallest previously published MLX tier.
Read this before downloading
This is a code-capable model with degraded factual recall. It is not a general-purpose K3.
Measured with greedy decoding, 40 new tokens:
| prompt | output | verdict |
|---|---|---|
def merge_intervals(intervals): | intervals = sorted(intervals, key=lambda x: x[0]) → merged = [] → for interval in intervals: → if not merged or merged[-1][1] < interval[0]: | correct |
The capital of France is | the capital of France is the capital of France is… | fails |
机器学习的基本原理是 | 理解数据，从数据中学习，学习是，学习，学习… (starts coherently, then loops) | degrades |
If you need factual recall or Chinese, use a larger tier. If you want a K3 that writes code on a 192 GB Mac, this is the only build we know of that fits.
| size | 181.2 GB (57 files) |
| experts | 160 of 896 kept per layer (82% pruned), 2-bit affine, group 128 |
| shared experts | both kept, never pruned |
| non-expert | per-component: attention 6-bit, embeddings/router 8-bit |
| load time | 53 s |
| decode | 3.93 tok/s (measured on a CUDA node; Apple Silicon will differ) |
| routing | top-16 of 160 = 10.0% density (base 1.8%) |
Why the factual recall is gone, precisely
A token's MoE output combines its top-k experts with gate weights renormalized to sum to 1, so
the model cannot down-weight an expert it dislikes. With p_j = P(expert j is in a token's top-k),
a kept set S serves Σ_{j∈S} p_j / top_k of the router's demand. We measured this on a 32k-token
calibration set:
| kept | random | AIMER (weight-only) | REAP (this build) | ideal |
|---|---|---|---|---|
| 160 | 17.9% | 22.3% | 51.7% | 54.3% |
| 376 | 42.0% | 48.2% | 77.5% | 79.6% |
At 160 experts a token reaches ~8.3 of the 16 experts it wanted. That is enough for code and not for facts. Recovering full capability needs ~77% retained mass ≈ 376 experts ≈ 366 GB, which does not fit 192 GB at MLX's 2-bit floor. So this limitation is arithmetic, not a tuning failure.
Things we tried that did not help, so you don't repeat them:
- Lower
top_k(16 → 12 → 8 → 6): the France prompt loops identically at every value, and Chinese gets worse. The damage is missing knowledge, not misweighted survivors. - A weight-only criterion (AIMER): only 1.25× better than random. At 137 experts it retained 19.4% and every prompt looped.
- Shared-basis + low-rank factorization (keep all 896 experts as
W_base + A_jB_j): closed by measurement — the shared mean alone has 99.9% relative error and rank 256 still 86%. K3's experts do not share a basis.
Usage
# pip install mlx mlx-lm
# this repo bundles kimi_k3.py; register it with mlx_lm first:
# cp kimi_k3.py $(python -c "import mlx_lm.models,os;print(os.path.dirname(mlx_lm.models.__file__))")/
from mlx_lm import load, generate
model, tokenizer = load("./Kimi-K3-mlx-reap160-2bit")
print(generate(model, tokenizer, "def merge_intervals(intervals):", max_tokens=128))
Raise the Metal wired limit — it defaults to ~75% of RAM, below this build:
sudo sysctl iogpu.wired_limit_mb=185000
What has and has not been checked
| check | status |
|---|---|
| structural: config, shard index, module coverage | done |
| loads and generates | done |
| code generation | done — correct |
| factual recall | done — fails |
| Chinese | done — degrades |
| perplexity / benchmarks | not done |
| Apple Silicon decode speed | not done (3.93 tok/s measured on a CUDA node) |
The prune plan and the REAP saliency ship with the weights, so the selection is auditable and the retained-mass numbers above are reproducible.
Attribution
- Base model:
moonshotai/Kimi-K3. - Loader and streaming converter vendored from PipeNetwork/kimi-k3-mlx.
- REAP expert saliency: arXiv 2510.13999.
- Prior art at 512 GB:
pipenetwork/Kimi-K3-REAP80-MLX-mxfp4-q8.