Kimi-K2.7-Code-Alis-MLX-Dynamic-3.6bpw

Part of the Kimi K2.7 Code · Alis MLX Dynamic collection.

[!IMPORTANT]

➡️ 가중치는 VLM 리포로 통합되었습니다 / Weights moved to the VLM repo

This repo kept a duplicate of the LLM tensors. The safetensors have been removed here to drop the second public copy. The language-model weights are byte-identical to the VLM build (VLM adds MoonViT + projector, +0.9 GB).

Download and run: avlp12/Kimi-K2.7-Code-Alis-MLX-Dynamic-3.6bpw-VLM

The rest of this card is kept as the text/code recipe and benchmark write-up.

A sensitivity-graded 3.6-bit MLX quantization of moonshotai/Kimi-K2.7-Code — a ~1T-parameter (32B active) DeepSeek-V3-style MoE coding model — built to run on Apple-Silicon M3 Ultra hardware. Shipped weights now live in the VLM sibling.

📘 This card documents the text / code recipe (mlx-lm, two-machine pipeline, mlx_lm.server). pipeline_tag remains text-generation.

For weights — text, code, image, or video — use Kimi-K2.7-Code-Alis-MLX-Dynamic-3.6bpw-VLM.

465 GB (433 GiB) on disk. It fits a single clean 512 GB M3 Ultra, and runs with huge headroom split across two 512 GB machines (≈233 GB per box) over Thunderbolt.

Why this build exists

Moonshot ships Kimi-K2.7-Code with its routed experts already INT4 (compressed-tensors, group-size 32, QAT) and everything else in bf16 — about 595 GB. The community MLX conversions either keep the experts at 4-bit and so need ~600–768 GB of memory (don't fit a single 512 GB box), or drop uniformly to ~3.5-bit.

This build takes a different route:

• Re-quantizes from the INT4 master correctly. mlx-lm mishandles compressed-tensors checkpoints during re-quantization (#907) — asking for 3-bit yields 4.99 bpw / 640 GB. We dequantize the packed INT4 experts in-memory before re-quantizing, so the experts actually land below 4-bit.
• Spends bits where they matter. Routed experts (≈99% of params) go to 3-bit; the residual-writing down_proj is upgraded to 4-bit on a spread of 16 layers (the most quantization-sensitive expert projection); attention (MLA), shared experts, the dense layer and the embeddings/head all stay at 6-bit; the MoE router stays bf16.

The result is a genuinely 512-GB-class build that keeps reasoning and coding quality intact.

Recipe (verified from the output config.json)

Effective 3.62 bits/weight. The down_proj-on-a-subset upgrade is the cheapest quality lever for a low-bit MoE — the residual-writing projection is by far the most sensitive expert matrix.

Memory & how to run it

Kimi-K2.7-Code uses Multi-head Latent Attention (MLA), so the KV cache is tiny — only the compressed latent (kv_lora_rank 512 + rope 64 = 576 values) per token per layer, ≈ 68.6 KB/token across all 61 layers.

The model's native context window is 256K tokens (262,144; YaRN-extended, rope_theta 50000). Even at the full 256K, weights + KV stay far under memory on either deployment.

Single clean 512 GB box — 465 GB weights leave ~47 GB; fine for inference, comfortable with int8 KV.

Two M3 Ultras over Thunderbolt (pipeline-parallel, recommended for headroom) — each machine loads only its layer-half (~233 GB), so neither box approaches its 512 GB limit, leaving room for other workloads.

# 2-machine pipeline-parallel (mlx-lm, ring backend over a Thunderbolt bridge)
mlx.launch --backend ring --hosts <ip0>,<ip1> python your_generate.py
# each rank: model, tok = mlx_lm.utils.pipeline_load("/path/to/this/model")

How it compares (Kimi-K2.7-Code MLX builds)

This build is the only one shipped with the experts re-quantized below their INT4 master via the #907 fix, a down_proj-protected sensitivity-graded recipe, and a verified two-machine pipeline path.

Benchmarks

Same harness as the reference runs (mlx_lm.perplexity on allenai/tulu-3-sft-mixture, seq 2048, 50 samples).

PPL is reported for build-vs-build comparison only — absolute perplexity is misleading at low bit-width. The KL divergence against a 4-bit reference of this same model (experts 4-bit g64, everything else 6-bit; the experts' native master is INT4, so 4-bit is effectively near-lossless for them) is the truer low-bit quality metric — it exposes distribution drift that PPL's averaging hides (Accuracy is Not All You Need). The picture is two-sided and worth stating honestly:

• On typical tokens the 3-bit experts cost almost nothing — the median per-token KL is 0.006 nats, and mean KL is only 0.076 across the ~90% of positions whose greedy top-1 is unchanged.
• But on ~10% of positions the greedy top-1 token flips vs the 4-bit reference, and those flips are mostly decisive, not coin-flips: at a flip the reference assigned its own pick ≈0.59 probability while this build's pick got ≈0.14 (only 15% of flips are near-ties). That tail (mean KL ≈1.29 on flipped positions) lifts the overall mean KL to 0.199 nats.

So: distributionally very close to a 4-bit build most of the time, with a real ~10% greedy-divergence cost for dropping the bulk of the experts to 3-bit — the expected price of fitting a ~1T model into 465 GB. PPL (3.735) also sits right next to far larger MLX MoE builds on the same harness.

What the 4-bit reference is — it's the original. The routed experts (≈99% of the weights) ship as a native INT4 master; there is no higher-precision source for them. So every 4/6/8-bit build sits at ≈ that master (you cannot beat INT4) and differs only in size — a 4-bit build is effectively the original model. This 3.6-bit build is therefore the one meaningful step below the original, and the KL 0.199 / 10.2 % flip above is its measured cost against the original itself, not against an arbitrary higher-bit sibling. Going to 6- or 8-bit would only grow the file, not the quality.

Correctness

• Real-weight reconstruction of the INT4 experts verified sane across early/mid/late layers (finite, expected magnitudes).
• Coherent generation in the distributed setup: correct iterative-Fibonacci and merge_intervals implementations (with passing asserts), correct technical Q&A. Kimi-K2.7-Code is a reasoning model and emits <think>…</think> before its answer.
• Per-machine peak memory in the 2-machine run: 233 GB / 226 GB.

Usage

This build needs an mlx-lm that loads compressed-tensors-derived Kimi (model_type: kimi_k25, DeepSeek-V3 engine). Use trust_remote_code=True (the tokenizer is tiktoken-based — pip install tiktoken blobfile).

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("avlp12/Kimi-K2.7-Code-Alis-MLX-Dynamic-3.6bpw",
                  trust_remote_code=True,
                  tokenizer_config={"trust_remote_code": True})
msgs = [{"role": "user", "content": "Write a Python LRU cache with O(1) get/put."}]
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True)
print(generate(model, tok, prompt=prompt, max_tokens=512, sampler=make_sampler(temp=0.0)))

Credits

• Base model: moonshotai/Kimi-K2.7-Code (Modified MIT).
• Built with mlx-lm on Apple MLX.

Citation

Kimi-K2.7-Code-MLX-3.6bpw — sensitivity-graded 3.6-bit MLX quantization of Kimi-K2.7-Code, 2026. Base model: Moonshot AI, Kimi-K2.7-Code.