kernelpool/Kimi-K3-2bit-UVMAX
Mixed-precision (UVMAX) quantization of moonshotai/Kimi-K3.
What is UVMAX?
UVMAX is a mixed-precision scheme: bit widths are assigned per tensor class from measured round-trip quantization error, rather than uniformly.
| Tensor class | Bits | Parameters | Size | Share |
|---|---|---|---|---|
| Expert FFNs (routed, latent space) | 2 (gs 128) | 2.72 T | 713.2 GiB | 93.8% |
| Shared experts, MoE latent projections, dense MLP | 8 (gs 64) | 17.5 B | 17.4 GiB | 2.3% |
| Attention (KDA + MLA, all projections) | 6 (gs 64) | 36 B | 27.4 GiB | 3.6% |
Embeddings, lm_head | 4 (gs 64) | 2.4 B | 1.2 GiB | 0.2% |
| MoE routers | 8 (gs 64) | 0.6 B | 0.6 GiB | 0.1% |
| Vision tower + projector (unquantized bf16) | — | 0.4 B | 0.8 GiB | 0.1% |
| Norms, AttnRes projections, gate params (unquantized) | — | — | 0.1 GiB | <0.1% |
Use with mlx
This model requires Kimi K3 support from mlx-lm PR #1626, which has not yet been merged. Until it is included in an mlx-lm release, install mlx-lm from the PR branch:
pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1626/head
pip install tiktoken
from mlx_lm import load, generate
model, tokenizer = load(
"kernelpool/Kimi-K3-2bit-UVMAX",
tokenizer_config={"trust_remote_code": True},
trust_remote_code=True,
)
prompt = "hello"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, verbose=True)