leonsarmiento/Orion-26B-A4B-v1-6bit-XL-mlx
This model was converted to MLX format from TheDrummer/Orion-26B-A4B-v1 using BaseQuant_XL 6/8-bit mixed quantization optimized for Apple Silicon. The vision encoder is preserved (full multimodal).
Orion is TheDrummer's finetune of google/gemma-4-26B-A4B-it — a 25.2B-parameter MoE (Mixture of Experts) model with 128 experts per layer (avg. 3.8B active parameters per token), tuned in his roleplay/creative-writing lineage. The unquantized BF16 source was used for maximum fidelity.
BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying aggressive quantization to the bulk parameters. tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).
About XL Quantization
BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.
Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.
Use with mlx
pip install -U mlx-vlm
python -m mlx_vlm.generate --model leonsarmiento/Orion-26B-A4B-v1-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"
BaseQuant_XL Quantization Strategy
| Bit Depth | Layers | Rationale |
|---|---|---|
| bf16 (unquantized) | router.proj (gate), mlp (shared/dense expert), vision tower | Routing decisions and the dense expert that processes every token — errors here cascade through all tokens |
| 8-bit | embed_tokens, self_attn | Every-token layers with moderate sensitivity — 8-bit is near-lossless |
| 6-bit | experts.switch_glu (routed experts) | Bulk of parameters — only a subset of experts active per token |
Quantization Details
| Layer | Bits | Group Size |
|---|---|---|
router.proj (gate) | bf16 | — |
mlp (shared/dense expert) | bf16 | — |
| Vision tower (full) | bf16 | — |
embed_tokens | 8 | 64 |
self_attn | 8 | 64 |
experts.switch_glu (routed experts) | 6 | 64 |
| Default fallback | 8 | 64 |
- Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
- Bits per weight: ~7.1
- Total size: 22.8 GB (5 shards)
- Group size: 64
- Method: Custom
quant_predicateviamlx_vlm - Note:
tie_word_embeddings=True— no separate lm_head
Recommended Inference Parameters
| Parameter | Value |
|---|---|
temperature | 1.0 |
top_p | 1.0 |
top_k | 64 |
min_p | 0.01 |
repeat_penalty | 1.1 |
Reasoning Parsing in LM Studio and Others
- Start string:
<|channel|>thought - End string:
<|channel|>
Chat Template
The chat template is synced to the canonical Google upstream version from google/gemma-4-26B-A4B-it/chat_template.jinja, replacing the template shipped with the source finetune. Do not rely on the HF cache copy — it may be stale.
Key markers in the latest template: preserve_thinking, continues_into_next, prev_non_tool_role, null handling in format_argument.
The template is injected into both chat_template.jinja (standalone file) and tokenizer_config.json (chat_template JSON field).