Kimi-K2.6 RAM GGUF — 447 GB (Q3_K experts)
Mixed-precision GGUF quantization of moonshotai/Kimi-K2.6, produced with the RAM (Resource-Aware Mixed-precision) pipeline.
Higher-quality variant. For a lower-footprint build, see baa-ai/Kimi-K2.6-RAM-344GB-GGUF.
File
| File | Size | Expert bits | Attention bits | Target hardware |
|---|---|---|---|---|
kimi-k2.6-ram-447gb.gguf | 447 GB | Q3_K | Q5_K–Q8_0 (probe-allocated) | 2× 256 GB or 1× 512 GB |
Method
Quantization bit depths are assigned per-tensor using sensitivity probing rather than a uniform scheme. Each attention tensor receives bits proportional to how much its output diverges under quantization noise, measured across 8 random probes. Expert tensors (384 routed experts × 60 MoE layers) are quantized uniformly at Q3_K.
Architecture: DeepSeek-V3 decoder (kimi_k2), 61 layers, 384 routed experts / 8 active per token, MLA attention, hidden_size=7168, 1.03T total parameters.
Usage (llama.cpp)
llama-cli \
-m kimi-k2.6-ram-447gb.gguf \
-c 8192 \
--temp 0.6 \
-p "You are a helpful assistant."
Notes
- No importance matrix (imatrix) was used — the 1.1 TB Q8_0 intermediate required for imatrix generation exceeds available RAM for inference on this machine. Bit allocation from sensitivity probing provides the primary quality signal.
- The original model uses Neural Magic compressed-tensors INT4 for expert weights. These are dequantized and re-quantized to GGUF format during conversion.