Qwen-35B-A3B-SignOfFour-Coder-GGUF

A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.

[!IMPORTANT] Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.

Contents

• Architectural Specifications
• Composition
• Merge Methodology
• Layer-Stratified Policies
• Chat Template
• Available quants
• Generation Parameters
• Lineage
• References

Architectural Specifications

Composition

Jackrong/Qwopus3.6-35B-A3B-Coder serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.

Merge Methodology

For each floating-point parameter, a task delta is computed per donor \(k\):

$$ \Delta_k = D_k - W_0 $$

DARE pruning. A Bernoulli mask at retention density \(p\) zeroes out low-magnitude updates; surviving values are rescaled by \(p^{-1}\):

$$ \tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p) $$

TIES sign election. A consensus sign \(\Gamma\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:

$$ \Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right) $$

$$ \Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}k) = \Gamma\right)}{\sum{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon} $$

Depth-scaled reconstruction. The merged weight is reconstructed as:

$$ W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}} $$

where the layer scaling factor \(\lambda(l)\) across decoder layer index \(l \in [0, 39]\) is defined as:

$$ \lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right) $$

This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \((l \in [12, 28])\).

Layer-Stratified Policies

• Router protection: Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
• DeltaNet stability: Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
• MTP removed: Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.

Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

Available quants

[!TIP] Need fast GPU inference? Running a 35B MoE with CPU offloading or on under-provisioned consumer RAM can result in low token throughput. If you lack a local 24 GB+ GPU and want faster inference, you can spin up an hourly instance on RunPod in one click to run llama-server with 100% layer offload. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

Affiliate Disclosure: If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder
    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
    └── donor: Qwen/Qwen-AgentWorld-35B-A3B

Citation & References

• Jackrong/Qwopus3.6-35B-A3B-Coder
• ornith-ai/Ornith-1.5-35B-A3B
• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}