Ornith-1.5-35B-A3B — XYZ Quantizations
Balanced GGUF quantizations of Ornith‑1.5‑35V (a 35B total / ~3B active hybrid SSM+attention Mixture‑of‑Experts model), built with the XYZ method: a balanced, MoE‑aware allocation in which attention layers and expert weights are kept at comparable precision rather than starving one for the other.
Highlights
- Balanced allocation — routed experts (≈93% of parameters) and attention are quantized to similar effective bit‑rates; no "maxed attention / starved experts" skew.
- Int8 dense core — all non‑expert projections (attention q/k/v/output, SSM out/gate, router gate) kept at Q8_0; token embedding and output head also Q8_0.
- BF16 invariants preserved at every level: SSM α/β scalars, attention k/v, router input — negligible cost, maximal quality.
- Imatrix‑guided low‑bit expert quantization (IQ2 / IQ3 / Q4_K tiers) for accuracy at small sizes.
Files
| File | Size | BPW |
|---|---|---|
| Ornith‑1.5‑35B‑A3B‑Q2.5‑XYZ.gguf | 11.2 GiB | 2.72 |
| Ornith‑1.5‑35B‑A3B‑Q3‑XYZ.gguf | 12.5 GiB | 3.02 |
| Ornith‑1.5‑35B‑A3B‑Q4‑XYZ.gguf | 17.1 GiB | 4.13 |
| Ornith‑1.5‑35B‑A3B‑Q5‑XYZ.gguf | 21.1 GiB | 5.10 |
| Ornith‑1.5‑35B‑A3B‑Q5.5‑XYZ.gguf | 23.7 GiB | 5.72 |
| Ornith‑1.5‑35B‑A3B‑Q6.5‑XYZ.gguf | 27.8 GiB | 6.71 |
| Ornith‑1.5‑35B‑A3B‑Q8‑XYZ.gguf | 37.2 GiB | 9.01 |
Usage (llama.cpp)
# example: run the Q4 build fully on GPU
./llama-cli -m Ornith-1.5-35B-A3B-Q4-XYZ.gguf -ngl 99 -c 8192
Running on limited VRAM (8 GB GPU + 16 GB RAM)
Ornith is a MoE model (~3B active of 35B total), so it runs well by keeping the big routed experts in system RAM and putting only attention / SSM / shared parts on the GPU. Pin the experts to CPU with --n-cpu-moe, use a modest context, and quantize the KV cache to Q4_0 to cut VRAM.
# ~8 GB VRAM target: routed experts in 16 GB RAM, attention+SSM on GPU, q4_0 KV cache
./llama-server -m Ornith-1.5-35B-A3B-Q2.5-XYZ.gguf \
-ngl 999 -ncmoe 30 \
-c 8192 -ctk q4_0 -ctv q4_0 \
--temp 0.6 --threads 4 --parallel 1
Notes:
-ncmoe Nkeeps the routed MoE experts on CPU (use a small value like30); only attention / SSM / shared-expert / embeddings go to the GPU. This is what makes an 8 GB card usable for a 35B model.-ctk q4_0 -ctv q4_0quantizes the K/V cache to Q4_0, roughly halving its VRAM footprint so longer contexts fit in 8 GB.- If it OOMs, lower
-c(e.g.4096) or increase -ncmoe value or use a smaller level; if you have headroom, raise-c.
Method / provenance
Built from the BF16 master (Ornith‑1.5‑35B‑BF16.gguf, 71 GB) using llama.cpp llama-quantize with per‑tensor‑type rules plus an importance matrix (blocks 0–39). The MTP/speculative block is quantized to Q4_K.