KAT-Coder-V2.5-Dev · REAP-50 — GGUF
GGUF quants of the 50%-REAP-pruned KAT-Coder-V2.5-Dev (Gated-DeltaNet + MoE
hybrid): 256 → 128 experts, ~19B parameters, no MTP head.
- Flagship (NVFP4A16, vLLM):
Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 - Pruned bf16 source:
Ttimms/KAT-Coder-V2.5-Dev-REAP-50-bf16 - Pipeline + benchmarks: https://github.com/t-timms/kat-coder-nvfp4
Base-model quality (NVFP4A16 quant, greedy, instruct): HumanEval+ ~90%, MBPP+ ~90%. These are single greedy draws; the accuracy harness has a measured run-to-run spread of 1.85–4.27 pp per task (2026-09-05), so read small differences as unresolved.
Architecture
graph TD
Base["Kwaipilot/KAT-Coder-V2.5-Dev<br/>Qwen3.5 MoE - 256 experts - ~69 GB bf16"]
subgraph Build ["Build - RTX 5070 Ti, SM120"]
REAP["REAP expert prune 50% 256 -> 128 experts + router-renorm fix"]
Strip["strip vision tower + 333 untrained tensors"]
Quant["NVFP4A16 quantize (weight-only, data-free, 82 s)"]
end
subgraph HF ["Published formats"]
A16["REAP-50-NVFP4A16 - 12.45 GiB (default, vLLM)"]
W4A4["REAP-50-NVFP4-W4A4 (native FP4)"]
GPTQ["REAP-50-NVFP4A16-GPTQ (null result, kept for transparency)"]
GGUF["REAP-50-GGUF (Q4_K_M / Q5_K_M / Q6_K / Q8_0)"]
BF16["REAP-50-bf16 (pruned source)"]
end
Bench["A16 single draws - HumanEval+ 89.0% - MBPP+ 90.5% - SWE-bench Verified 52.0%"]
Base --> REAP --> Strip --> Quant --> A16
Strip --> BF16
BF16 -. re-quant .-> W4A4
BF16 -. re-quant .-> GPTQ
BF16 -. convert .-> GGUF
A16 --> Bench
Download
Grab one file, not the whole repo.
| File | Quant | Size | Notes |
|---|---|---|---|
kat-reap50-Q4_K_M.gguf | Q4_K_M | 11.4 GB | recommended — fits 16 GB with room for context |
kat-reap50-Q5_K_M.gguf | Q5_K_M | 13.3 GB | higher quality, tighter KV budget |
kat-reap50-Q6_K.gguf | Q6_K | 15.3 GB | near-lossless; little headroom on 16 GB |
kat-reap50-Q8_0.gguf | Q8_0 | 19.7 GB | needs >16 GB or CPU offload |
MoE with ~3B active parameters — CPU / Apple-Silicon inference is fast for the size.
For full precision, re-quantizing, or merging, use the
pruned bf16 source
and run convert_hf_to_gguf.py --no-mtp yourself.
Run it in
- llama.cpp —
llama-server -m <file>.gguf -ngl 99 - LM Studio (bundles a recent llama.cpp)
- koboldcpp
- Jan
- text-generation-webui
- Ollama — once its bundled llama.cpp includes this arch (see Requirements)
Measured on an RTX 5070 Ti (16 GB)
Measured 2026-09-24 on kat-reap50-Q4_K_M.gguf only (the other quants were not run). Plain decoding: this
model has no MTP head, so there is no speculative decoding. Hardware: RTX 5070 Ti (16,303 MiB), Ryzen 5 7600,
WSL2, with the desktop still using 1.3-1.9 GB of VRAM. Software: llama.cpp commit 3173a56 (source build,
2026-08-29).
llama-server --model kat-reap50-Q4_K_M.gguf -c 32768 -ngl 999 -fa on -np 1 \
--cache-type-k q8_0 --cache-type-v q8_0 -b 2048 -ub 512 --jinja
| Metric | Result |
|---|---|
| Decode, empty context (T = 0 / 0.6 / 1.0) | 181.0 / 183.6 / 183.5 tok/s |
| Decode at ~24K tokens of context (T = 0.6) | 151.9 tok/s |
| Prefill, ~24K-token prompt | ~4.7K tok/s (about 5 s) |
| Peak VRAM at 32K context, q8_0 KV | 12,873 MiB of 16,303 MiB (includes the desktop's 1.3-1.9 GB) |
Decode figures are medians of the server-reported predicted_per_second: 5 prompt types x 3 repeats at empty
context, 2 prompt types x 2 repeats at ~24K context. One machine and one build, so treat these as a reference
point rather than a guarantee.
Requirements
Needs a recent llama.cpp (the qwen35moe hybrid arch). Verified with
llama-server from a source build. Ollama support pending its bundled-llama.cpp bump.
llama-server -m kat-reap50-Q4_K_M.gguf -ngl 99 --port 8080
The chat template defaults to thinking on; pass
"chat_template_kwargs": {"enable_thinking": false} for direct output.
License
Apache-2.0 (inherits from Kwaipilot/KAT-Coder-V2.5-Dev). Pruning: REAP
(github.com/CerebrasResearch/reap, with a router-renormalization fix).