Triumvirate (GGUF)

This repository provides quantized GGUF binaries of pragmaticcs/Triumvirate, a 35B MoE (~3.1B active parameters per token) merge built for agentic coding, terminal tool use, and multi-turn repository workflows.

It pairs the strict patch syntax and low tool-error rate of KAT-Coder-V2.5-Dev with algorithmic reasoning from Ornith-1.5 and environment state tracking from Qwen-AgentWorld. Fused via depth-modulated DARE-TIES, it focuses on generating clean search/replace diffs, parsing terminal error feedback, and staying on-task inside autonomous agent loops without conversational drift.

[!TIP] Intended Use: Optimized for agent harnesses. It prioritizes valid tool invocations, concise step-by-step reasoning, and exact code modifications over conversational filler.

Contents

• Available Quants
• How to Run
• Recommended Generation Parameters
• Architectural Specifications
• Composition & Triumvirate Roles
• Merge Methodology
• Depth Modulation Dynamics
• Layer-Stratified Policies
• Chat Template
• Lineage
• Citation & References

Available Quants

[!TIP] Need fast GPU inference? Running a 35B MoE with CPU offloading or on under-provisioned consumer RAM can result in low token throughput. If you lack a local 24 GB+ GPU and want faster inference, you can spin up an hourly instance on RunPod in one click to run llama-server with 100% layer offload. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

Affiliate Disclosure: If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

Architectural Specifications

Composition & Triumvirate Roles

Merge Methodology

For each parameter tensor, fine-tuning task deltas are computed relative to the base anchor:

$$ \Delta_k = D_k - W_0, \quad k \in {o, a} $$

1. Shape-Safe DARE Sparsification

For prunable weight matrices (numel > 4096), a Bernoulli mask at drop rate \(p_{\text{drop}}\) removes low-magnitude interference while rescaling retained coordinates to preserve expectation:

$$ \tilde{\Delta}k = \frac{1}{1 - p{\text{drop}}} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(1 - p_{\text{drop}}) $$

2. TIES Sign Election with Multi-Donor Variance Damping

A sign consensus direction \(\Gamma\) is computed across surviving updates:

$$ \Gamma = \operatorname{sgn}\left(\sum_k \lambda_k \tilde{\Delta}_k\right) $$

Conflicting updates are masked to zero. Coordinates where both donors agree receive a dampening factor of \(\gamma = 1.25\) to prevent activation explosion:

$$ \Delta_{\text{consensus}} = \frac{\sum_k \left(\lambda_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)\right)}{\mathbb{I}\left(\sum \text{agrees} > 1\right) \cdot (\gamma - 1) + 1} $$

$$ W_{\text{final}} = W_0 + \Delta_{\text{consensus}} $$

3. Calibrated MoE Router Gate Harmonization

Router gates (mlp.gate.weight) are merged without DARE sparsification, scaling interpolated matrices to maintain the base model's Frobenius norm:

$$ \tilde{W}_k = W_k \cdot \frac{|W_0|_F}{|W_k|_F + \epsilon} $$

$$ W_{\text{gate}} = \alpha_b W_0 + \alpha_o \tilde{W}_o + \alpha_a \tilde{W}_a, \quad \text{where } \alpha_k = 0.35 \lambda_k(l), ; \alpha_b = \max\left(0, 1 - (\alpha_o + \alpha_a)\right) $$

$$ W_{\text{gate, final}} = W_{\text{gate}} \cdot \frac{|W_0|F}{|W{\text{gate}}|_F + \epsilon} $$

Depth Modulation Dynamics

Uniform task addition across all layers disrupts low-level tokenization and high-level output selection. Layer scaling factors \(\lambda_o(l)\) and \(\lambda_a(l)\) are modulated continuously across decoder layers \(l \in [0, 39]\) using normalized depth \(t = \frac{l}{39} \in [0, 1]\):

$$ \lambda_o(l) = 0.15 + 0.50 \cdot \sin\left(\pi \cdot t^{0.9}\right) $$

$$ \lambda_a(l) = \begin{cases} 0.25 \cdot \sin^2\left(\pi \frac{t - 0.35}{0.50}\right) & \text{for } 0.35 \le t \le 0.85 \ 0.0 & \text{otherwise} \end{cases} $$

To prevent the base coding anchor from being diluted, if \(\lambda_o + \lambda_a > 0.72\), both donor scales are proportionally normalized to guarantee a minimum base anchor weight of \(\lambda_k \ge 0.28\).

Depth Band Allocation

Layer-Stratified Policies

• Recurrent State Integrity: DeltaNet's temporal convolution (conv1d) and update gates (in_proj_ba) are strictly excluded from DARE dropout (zero dropout, \(p_{\text{drop}} = 0.0\)) to eliminate numerical divergence in the linear attention state space.
• Vocabulary Padding Hygiene: Unused trailing vocabulary rows are clamped using nan_to_num to prevent uninitialized padding noise from contaminating token embeddings.

Chat Template

This model includes the Improved Chat Template for Qwen 3.x by Olivia Rossi, supporting:

• Dedicated Chain-of-Thought (<thought>...</thought>) reasoning blocks.
• Multi-turn tool calling with strict schema verification.
• Built-in error-recovery heuristics for malformed tool parameters.
• Whitespace trimming and prompt token-waste minimization.

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Triumvirate
    ├── Base Anchor:       Kwaipilot/KAT-Coder-V2.5-Dev
    ├── Reasoning Donor:   ornith-ai/Ornith-1.5-35B-A3B
    ├── Simulation Donor:  Qwen/Qwen-AgentWorld-35B-A3B
    └── Quantization:      pragmaticcs/Triumvirate-GGUF

Citation & References

• Kwaipilot/KAT-Coder-V2.5-Dev
• ornith-ai/Ornith-1.5-35B-A3B
• Qwen/Qwen-AgentWorld-35B-A3B
• pragmaticcs/Triumvirate (Safetensors Base)
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}