UltiMerge

[!IMPORTANT] Pure text-causal build. Multi-token-prediction heads and multimodal vision weights were decoupled and stripped before merging. That drops roughly 4.2 GB of VRAM overhead and gives immediate out-of-the-box compatibility with vLLM, SGLang, and llama.cpp (GGUF).

Contents

• Overview
• Quad-DNA Architecture
• The GS-MoE Transformation Pipeline
• Mathematical Formulation
• Inside the Merged Weight Manifold
• Deployment & Inference
• Recommended Sampling
• Model Specifications
• Attributions & Citations

Overview

UltiMerge is a 35B-A3B mixture-of-experts model built on the Qwen 3.6 hybrid architecture — 30 Gated DeltaNet linear-recurrence layers plus 10 full GQA attention layers, with 256 routed experts (top-8 active) behind one always-on shared expert.

It isn't a fine-tune. It's a geometric merge of four independently trained checkpoints, each strong in a different discipline: world simulation and tool use, repository-scale software engineering, deep chain-of-thought reasoning, and adversarial self-correction. A five-stage pipeline — Geometric Spectral MoE (GS-MoE) — isolates what each donor actually learned, filters out merge noise, and extrapolates the combination along the checkpoints' shared weight manifold, rather than simply averaging them.

The result runs with the compute footprint of a 3.1B-parameter model while carrying the combined skill set of four 35B-class specialists.

Quad-DNA Architecture

Source checkpoints

Each checkpoint's task vector is isolated relative to the shared Qwen 3.6 base before any merge operation runs, which is what keeps the fusion from collapsing into an average of four unrelated skills.

The GS-MoE Transformation Pipeline

Recurrence parameters bypass low-rank truncation entirely (stage 01) to avoid degrading memory behavior at long context lengths — everything downstream of that is where the actual skill-blending happens.

Mathematical Formulation

1. Invariant SSM layer preservation. Recurrence parameters are simple-averaged across donors to protect linear-attention stability up to 256K context:

$$\mathbf{W}{\text{SSM}} = \frac{1}{N} \sum{i=1}^N \mathbf{W}_i$$

2. MoE router entropy-locked alignment. Merged gate weights are rescaled against the anchor's Frobenius norm to stop routing entropy from exploding:

$$\mathbf{W}{\text{gate}} = \bar{\mathbf{W}}{\text{gate}} \cdot \frac{|\mathbf{W}_0|F}{|\bar{\mathbf{W}}{\text{gate}}|_F + \epsilon}$$

3. DARE subspace noise filtering. Each expert's task delta $\Delta \mathbf{W}_i = \mathbf{W}_i - \mathbf{W}_0$ is sparsified at Bernoulli probability $p = 0.20$ and rescaled:

$$\widetilde{\Delta \mathbf{W}}_i = \frac{\Delta \mathbf{W}_i \odot \mathbf{M}}{1 - p}, \quad \mathbf{M} \sim \text{Bernoulli}(1 - p)$$

4. Spectral truncation & rescaling (STAR). Randomized low-rank SVD filters high-frequency merge noise while retaining $\gamma = 0.90$ of total Frobenius energy:

$$\mathbf{\Sigma}'{:k} = \mathbf{\Sigma}{:k} \cdot \frac{|\mathbf{\Sigma}|F}{|\mathbf{\Sigma}{:k}|F + \epsilon}, \qquad \widehat{\Delta \mathbf{W}}i = \mathbf{U}{:k} \mathbf{\Sigma}'{:k} \mathbf{V}_{:k}^T$$

5. Model Stock hyperspherical extrapolation. The merged update is extrapolated along the intersection manifold using the optimal cosine-angle factor $t^*$:

$$t^* = \frac{N \cos \bar{\theta}}{1 + (N - 1)\cos \bar{\theta}}, \qquad \mathbf{W}_{\text{final}} = \mathbf{W}0 + t^* \cdot \frac{1}{N}\sum{i=1}^N \widehat{\Delta \mathbf{W}}_i$$

Inside the Merged Weight Manifold

Deployment & Inference

1. Hugging Face transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliviaRossi/UltiMerge"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are UltiMerge, an elite autonomous software engineer and reasoning system."},
    {"role": "user", "content": "Implement a thread-safe, high-concurrency LRU cache in Python using doubly linked lists and hash maps, complete with unit tests."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.2,
    top_p=0.9,
    repetition_penalty=1.05
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

2. Production server with vLLM

vllm serve OliviaRossi/UltiMerge \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --port 8000

3. Local GGUF inference (llama.cpp)

# 1. Convert to BF16 GGUF
python3 convert_hf_to_gguf.py OliviaRossi/UltiMerge \
  --outfile ./ultimerge_35b_f16.gguf \
  --outtype bf16

# 2. Quantize to Q4_K_M
llama-quantize ./ultimerge_35b_f16.gguf ./ultimerge_35b_Q4_K_M.gguf Q4_K_M

# 3. Launch interactive CLI
llama-cli \
  -m ./ultimerge_35b_Q4_K_M.gguf \
  -p "<|im_start|>system\nYou are UltiMerge.<|im_end|>\n<|im_start|>user\nWrite a lock-free queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
  -n 2048 \
  --temp 0.2 \
  -ngl 99

4. Ollama Modelfile

FROM ./ultimerge_35b_Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM "You are UltiMerge, an elite software engineering agent and reasoning system."

PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

Recommended Sampling

Model Specifications

Attributions & Citations

Source checkpoints

• Alibaba Qwen Team — Qwen-AgentWorld-35B-A3B, Language Models as Interactive World Simulators
• Kwaipilot Team — KAT-Coder-V2.5-Dev, Autonomous Software Engineering with MoE Foundations
• Jackrong — Qwopus3.6-35B-A3B-Coder, Opus-Grade Reasoning Distillation in MoE Models
• Ornith AI — Ornith-1.5-35B-A3B, Recursive Self-Correction and Constraint Alignment

@article{qwen_agentworld_2026,
  title        = {Qwen-AgentWorld: Language Models as Interactive World Simulators},
  author       = {{Qwen Team, Alibaba Group}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B}
}

@article{kwaipilot_kat_coder_2026,
  title        = {KAT-Coder: Autonomous Software Engineering with Mixture-of-Experts},
  author       = {{Kwaipilot Team}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev}
}

@article{jackrong_qwopus_2026,
  title        = {Qwopus-3.6: Distilling Opus-Grade Reasoning into High-Throughput MoE Architectures},
  author       = {Jackrong},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder}
}

@article{ornith_ai_2026,
  title        = {Ornith-1.5: Recursive Self-Correction and Constraint Alignment in Hybrid Models},
  author       = {{Ornith AI}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B}
}

@misc{ultimerge_2026,
  title        = {UltiMerge: A Geometric Spectral MoE Fusion of Four 35B-A3B Specialists},
  author       = {Rossi, Olivia},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/OliviaRossi/UltiMerge}
}