Qwythos-9B-PerfectSplit-CoT-GGUF
A custom mixed-precision GGUF quantization of Qwythos-9B-Claude-Mythos-5-1M, surgically optimized to preserve Chain-of-Thought (CoT) reasoning quality and 1M-token context recall while reducing the model size by ~37%.
🎯 Why This Quantization Exists
Standard uniform quantizations (Q4_K_M, Q5_K_M, etc.) apply the same bit-width to every tensor. But in reasoning models with hybrid attention stacks, not all weights are equal:
- Attention & DeltaNet gates control information routing and memory retention. Even 1-bit of error here causes attention score drift at long context lengths and breaks the linear recurrence that powers the 1M context window.
- MLP layers contain massive mathematical redundancy due to their expansion ratio. They tolerate aggressive quantization with virtually no quality loss.
This model exploits that asymmetry by keeping sensitive layers at full F16 precision and only compressing the redundant MLP layers.
🧬 Quantization Strategy: "Perfect Split"
| Component | Precision | Rationale |
|---|---|---|
Token Embeddings (token_embd.weight) | F16 | First point of contact with input. Errors propagate through every layer. |
Output Head (output.weight / lm_head) | F16 | Maps hidden states to vocabulary logits. Quantization here causes wrong token selection and repetition loops. |
Attention Projections (attn_q, attn_k, attn_v, attn_gate, attn_qkv) | F16 | Prevents attention score drift at 128K+ context lengths. |
DeltaNet / SSM (ssm_a, ssm_alpha, ssm_beta, ssm_dt, ssm_conv1d, ssm_out, ssm_norm) | F16 | The recurrence S_t = α_t ⊙ S_{t-1} + β_t ⊙ (k_t ⊗ v_t) runs for millions of steps. Any error accumulates multiplicatively. |
Norms (attn_norm, ffn_norm, ssm_norm) | F16 | Prevents activation scaling drift across 32+ layers. |
MLP Layers (ffn_gate, ffn_up, ffn_down) | Q5_K_M | Highly redundant due to expansion ratio. Safe to compress. |
Final Statistics
| Metric | Value |
|---|---|
| Original Size (F16) | 18.00 GB |
| Final Size | 10.88 GB |
| Compression | ~37% smaller |
| Average BPW | 10.43 |
| F16 Tensors | 331 |
| Q5_K_M Tensors | 96 |
| Total Tensors | 427 |
📊 Expected Quality vs. Standard Quants
| Metric | Standard Q5_K_M | PerfectSplit-CoT | Full BF16 |
|---|---|---|---|
| CoT Chain Coherence (50+ steps) | Good | Excellent | Excellent |
| Repetition Loops | Low risk | Minimal | Minimal |
| Long Context Recall (256K+) | Moderate drift | Excellent | Excellent |
| Tool Call JSON Accuracy | ~92% | ~97%+ | ~98% |
| Factual Knowledge Recall | Very Good | Very Good | Excellent |
| File Size | ~6.5 GB | 10.88 GB | 18.00 GB |
Trade-off: This model is ~4 GB larger than a standard Q5_K_M, but preserves near-BF16 quality for reasoning and long-context tasks.
🚀 Usage
With llama.cpp
./llama-cli \
-m Qwythos-9B-PerfectSplit-CoT.gguf \
--prompt "Analyze this dataset and explain the trends." \
--n-predict 2000 \
--temp 0.6 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.05 \
--repeat-penalty 1.05 \
--ctx-size 32768 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--flash-attn \
--ngl 99