Kimi-K3-pruned75
This is a calibration-free, safetensors-only expert-pruned checkpoint derived from moonshotai/Kimi-K3. It removes 75% of the routed experts and is intended for vLLM serving and evaluation smoke tests while accurate calibration-based pruning, such as REAP support in llm-compressor, is under development.
This is an experimental prototype, not a REAP-pruned checkpoint. Expect a large quality regression relative to the source model.
Pruning
- Source experts per MoE layer: 896
- Retained experts per MoE layer: 224 (25% retained, 75% removed)
- MoE layers: 92
- Ranking: descending signed
sum(router.weight[i]), independently per layer - Removed data: pruned expert tensors and their corresponding router rows
- Router and expert indices were compacted, and the config was updated to 224 experts
- Calibration data: none
- Retained expert count is divisible by TP=4 (56 experts per rank)
- Tensor payload: 475,131,719,168 bytes (442.50 GiB)
- Indexed weights: 126,276 across 96 safetensors shards
pruning_manifest.json records the source hashes, exact per-layer expert
remap, pruning method, and output statistics. The checkpoint was validated
against every shard keyset and payload size, router and bias permutation,
sampled expert tensors in all layers, and auxiliary-file hashes. Each retained
224-expert set is the exact prefix of the corresponding ranking used for
mgoin/Kimi-K3-pruned50.
Validated vLLM run
The checkpoint loaded and served successfully with the default vLLM loader
(no --load-format) on one node with four NVIDIA GB300 GPUs:
- vLLM:
0.26.1rc1.dev142+g30b4e7f47 - vLLM commit:
30b4e7f479674a9c4d8889d4857294d3bd5e6849 - FlashInfer:
0.6.16rc5 - Hardware: 4x NVIDIA GB300, 284208 MiB each
- Parallelism: TP=4 on one node
- Weight load time: 210.80 seconds
- Model memory: 119.06 GiB per GPU
- Available KV cache: 132.14 GiB per GPU
- KV-cache capacity: 7,615,283 tokens
- Steady GPU allocation: approximately 262.90 GiB per GPU
vllm serve mgoin/Kimi-K3-pruned75 \
--served-model-name kimi-k3-pruned-75pct \
--host 0.0.0.0 \
--port 18082 \
--tensor-parallel-size 4 \
--trust-remote-code \
--moe-backend auto \
--gpu-memory-utilization 0.95 \
--max-model-len 32768 \
--kv-cache-dtype fp8 \
--attention-backend FLASHINFER_MLA \
--attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
--enable-prefix-caching \
--no-enable-flashinfer-autotune \
--max-num-seqs 8 \
--max-num-batched-tokens 8192 \
--max-cudagraph-capture-size 8
The health endpoint returned HTTP 200 and an arithmetic smoke test passed. Kimi K3 latent-MoE tail fusion reported that TP=4 is unsupported and automatically used the default path; serving remained healthy.
GSM8K
The evaluation used vLLM's standalone GSM8K harness against the
/v1/completions endpoint with the full test split:
.venv/bin/python tests/evals/gsm8k/gsm8k_eval.py \
--host http://${HEAD_NODE} \
--port 18082 \
--num-questions 1319 \
--num-shots 5 \
--max-tokens 256 \
--temperature 0 \
--seed 42
| Checkpoint | Accuracy | Correct | Invalid | Latency | Questions/s |
|---|---|---|---|---|---|
| Kimi-K3-pruned75 | 34.4200% | 454/1319 | 4/1319 | 536.326 s | 2.459 |
| Kimi-K3-pruned50 | 68.2335% | 900/1319 | 3/1319 | 481.863 s | 2.737 |
| Source Kimi-K3 | 95.5269% | 1260/1319 | 1/1319 | 530.362 s | 2.487 |
The pruned75 checkpoint regressed by 61.1069 percentage points relative to the source model and 33.8135 points relative to pruned50. All runs used the same 1,319-question test set, 5-shot prompt builder, 256-token limit, temperature, seed, vLLM checkout, and GB300 hardware.
License
This derivative is distributed under the upstream
Kimi K3 License.
See LICENSE in this repository.