Qwen-35B-A3B-SignOfFour-Coder

A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.

[!IMPORTANT] Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.

Contents

• Architectural Specifications
• Composition
• Merge Methodology
• Layer-Stratified Policies
• Chat Template
• Generation Parameters
• How to Use
• Lineage
• References

Architectural Specifications

Composition

Jackrong/Qwopus3.6-35B-A3B-Coder serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.

Merge Methodology

For each floating-point parameter, a task delta is computed per donor \(k\):

$$ \Delta_k = D_k - W_0 $$

DARE pruning. A Bernoulli mask at retention density \(p\) zeroes out low-magnitude updates; surviving values are rescaled by \(p^{-1}\):

$$ \tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p) $$

TIES sign election. A consensus sign \(\Gamma\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:

$$ \Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right) $$

$$ \Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}k) = \Gamma\right)}{\sum{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon} $$

Depth-scaled reconstruction. The merged weight is reconstructed as:

$$ W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}} $$

where the layer scaling factor \(\lambda(l)\) across decoder layer index \(l \in [0, 39]\) is defined as:

$$ \lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right) $$

This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \((l \in [12, 28])\).

Layer-Stratified Policies

• Router protection: Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
• DeltaNet stability: Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
• MTP removed: Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.

Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

How to Use

[!TIP] Deploy with Cloud Compute: If your local machine lacks sufficient VRAM to run this model at full precision, you can launch an on-demand GPU instance on RunPod. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

Affiliate Disclosure: If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

vLLM

vllm serve pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-reasoning \
  --reasoning-parser qwen3

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a precise agentic software engineer. Solve problems concisely."},
    {"role": "user", "content": "Write an asynchronous Python queue consumer with retry backoff."}
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(
    inputs,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    min_p=0.01,
    do_sample=True,
)

print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder
    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
    └── donor: Qwen/Qwen-AgentWorld-35B-A3B

Citation & References

• Jackrong/Qwopus3.6-35B-A3B-Coder
• ornith-ai/Ornith-1.5-35B-A3B
• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}