Triumvirate

Triumvirate is a 35B MoE (~3.1B active parameters per token) merge built for agentic coding, terminal tool use, and multi-turn repository workflows.

It pairs the strict patch syntax and low tool-error rate of KAT-Coder-V2.5-Dev with algorithmic reasoning from Ornith-1.5 and environment state tracking from Qwen-AgentWorld. Fused via depth-modulated DARE-TIES, it focuses on generating clean search/replace diffs, parsing terminal error feedback, and staying on-task inside autonomous agent loops without conversational drift.

[!TIP] Intended Use: Optimized for agent harnesses. It prioritizes valid tool invocations, concise step-by-step reasoning, and exact code modifications over conversational filler.

Contents

• Architectural Specifications
• Composition & Triumvirate Roles
• Merge Methodology
• Depth Modulation Dynamics
• Layer-Stratified Policies
• Chat Template
• Recommended Generation Parameters
• How to Use
• Lineage
• Citation & References

Architectural Specifications

Composition & Triumvirate Roles

Merge Methodology

For each parameter tensor, fine-tuning task deltas are computed relative to the base anchor:

$$ \Delta_k = D_k - W_0, \quad k \in {o, a} $$

1. Shape-Safe DARE Sparsification

For prunable weight matrices (numel > 4096), a Bernoulli mask at drop rate \(p_{\text{drop}}\) removes low-magnitude interference while rescaling retained coordinates to preserve expectation:

$$ \tilde{\Delta}k = \frac{1}{1 - p{\text{drop}}} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(1 - p_{\text{drop}}) $$

2. TIES Sign Election with Multi-Donor Variance Damping

A sign consensus direction \(\Gamma\) is computed across surviving updates:

$$ \Gamma = \operatorname{sgn}\left(\sum_k \lambda_k \tilde{\Delta}_k\right) $$

Conflicting updates are masked to zero. Coordinates where both donors agree receive a dampening factor of \(\gamma = 1.25\) to prevent activation explosion:

$$ \Delta_{\text{consensus}} = \frac{\sum_k \left(\lambda_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)\right)}{\mathbb{I}\left(\sum \text{agrees} > 1\right) \cdot (\gamma - 1) + 1} $$

$$ W_{\text{final}} = W_0 + \Delta_{\text{consensus}} $$

3. Calibrated MoE Router Gate Harmonization

Router gates (mlp.gate.weight) are merged without DARE sparsification, scaling interpolated matrices to maintain the base model's Frobenius norm:

$$ \tilde{W}_k = W_k \cdot \frac{|W_0|_F}{|W_k|_F + \epsilon} $$

$$ W_{\text{gate}} = \alpha_b W_0 + \alpha_o \tilde{W}_o + \alpha_a \tilde{W}_a, \quad \text{where } \alpha_k = 0.35 \lambda_k(l), ; \alpha_b = \max\left(0, 1 - (\alpha_o + \alpha_a)\right) $$

$$ W_{\text{gate, final}} = W_{\text{gate}} \cdot \frac{|W_0|F}{|W{\text{gate}}|_F + \epsilon} $$

Depth Modulation Dynamics

Uniform task addition across all layers disrupts low-level tokenization and high-level output selection. Layer scaling factors \(\lambda_o(l)\) and \(\lambda_a(l)\) are modulated continuously across decoder layers \(l \in [0, 39]\) using normalized depth \(t = \frac{l}{39} \in [0, 1]\):

$$ \lambda_o(l) = 0.15 + 0.50 \cdot \sin\left(\pi \cdot t^{0.9}\right) $$

$$ \lambda_a(l) = \begin{cases} 0.25 \cdot \sin^2\left(\pi \frac{t - 0.35}{0.50}\right) & \text{for } 0.35 \le t \le 0.85 \ 0.0 & \text{otherwise} \end{cases} $$

To prevent the base coding anchor from being diluted, if \(\lambda_o + \lambda_a > 0.72\), both donor scales are proportionally normalized to guarantee a minimum base anchor weight of \(\lambda_k \ge 0.28\).

Depth Band Allocation

Layer-Stratified Policies

• Recurrent State Integrity: DeltaNet's temporal convolution (conv1d) and update gates (in_proj_ba) are strictly excluded from DARE dropout (zero dropout, \(p_{\text{drop}} = 0.0\)) to eliminate numerical divergence in the linear attention state space.
• Vocabulary Padding Hygiene: Unused trailing vocabulary rows are clamped using nan_to_num to prevent uninitialized padding noise from contaminating token embeddings.

Chat Template

This model includes the Improved Chat Template for Qwen 3.x by Olivia Rossi, supporting:

• Dedicated Chain-of-Thought (<thought>...</thought>) reasoning blocks.
• Multi-turn tool calling with strict schema verification.
• Built-in error-recovery heuristics for malformed tool parameters.
• Whitespace trimming and prompt token-waste minimization.

Recommended Generation Parameters

How to Use

[!TIP] Deploy with Cloud Compute: If your local environment lacks sufficient VRAM to run a 35B model at full precision, you can deploy an on-demand GPU instance on RunPod. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

Affiliate Disclosure: If you register an account and load funds using this link, I receive platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

vLLM

Run with --language-model-only to bypass stripped multimodal vision components:

vllm serve pragmaticcs/Triumvirate \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.95 \
  --language-model-only \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-reasoning \
  --reasoning-parser qwen3

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/Triumvirate"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "system", 
        "content": "You are a software engineer and agentic problem solver. Think step-by-step in <thought> tags before answering."
    },
    {
        "role": "user", 
        "content": "Write a high-performance Python connection pool for an asynchronous Redis cluster with backoff retries."
    }
]

inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

output = model.generate(
    inputs,
    max_new_tokens=2048,
    temperature=0.55,
    top_p=0.95,
    min_p=0.01,
    do_sample=True,
)

print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Triumvirate
    ├── Base Anchor:       Kwaipilot/KAT-Coder-V2.5-Dev
    ├── Reasoning Donor:   ornith-ai/Ornith-1.5-35B-A3B
    └── Simulation Donor:  Qwen/Qwen-AgentWorld-35B-A3B

Citation & References

• Kwaipilot/KAT-Coder-V2.5-Dev
• ornith-ai/Ornith-1.5-35B-A3B
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}