📐 Qwen2.5-R1-Minny-1.5B-v2
Qwen2.5-R1-Minny-1.5B-v2 is a highly experimental, mathematically optimized 1.5B parameter Small Language Model (SLM). It represents a completely custom, from-scratch implementation of a novel model merging paradigm: Sens-Stock Fusion.
By bridging the microscopic, gradient-based layer routing of Sens-Merging (Liu et al., 2025) with the macroscopic, flat-minimum geometric projection of Model Stock (Jang et al., 2024), this model achieves a Pareto-optimal fusion of pure mathematical reasoning, syntax generation, and conversational instruction-following.
🔬 The "Golden Triangle" Topology
This model was constructed without backpropagation by merging three highly specialized, homologous fine-tunes anchored to a foundational reasoning base:
- Anchor (Base):
deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B - Expert A (Code):
agentica-org/DeepCoder-1.5B-Preview - Expert B (Math):
RLinf/RLinf-math-1.5B - Expert C (Chat/Structural):
mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-1.5B-v1.1
🧮 The Sens-Stock Methodology
Standard model merging techniques (like SLERP or TIES) suffer from uniform coefficient assignments and arbitrary magnitude scaling. Sens-Stock solves both the directional and magnitude problems of task vector arithmetic mathematically:
- Sens-Merging (The Direction): Using custom calibration subsets (
MATH-500Level 4/5,LeetCodeDataset, andMixture-of-Thoughts), we extracted the first-order Taylor expansion approximations of parameter sensitivity via backpropagation. We combined this with $L_2$ cross-task logit alignment to generate temperature-scaled Softmax routing coefficients ($\sigma_i^l$) for every parameterized layer $l$. This guarantees that the Code model dominates syntax layers while the Math model dominates logical reasoning layers. - Model Stock (The Magnitude): Rather than uniformly scaling the routed task vector, we calculated the geometric angle ($\theta$) between the fine-tuned task vectors in high-dimensional space. We applied the Model Stock projection ratio $t = \frac{N \cos \theta}{1 + (N - 1)\cos \theta}$ (where $N=3$) to dynamically project the Sens-Merged vector to the boundary of the flat-minimum loss basin.
Final Fusion Equation: $$W_{final}^l = W_{base}^l + t^l \cdot \sum_{i=1}^{N} \sigma_i^l \cdot (W_{expert_i}^l - W_{base}^l)$$
💻 Usage & Prompt Format
Because this model is anchored to the DeepSeek-R1 distillation lineage, it utilizes the Qwen ChatML format and inherently relies on Chain-of-Thought (CoT) <think> tags for complex reasoning.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "ForSureTesterSim/Qwen2.5-R1-Minny-1.5B-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
prompt = """Solve the following problem using Python.
First, explain the mathematical theory behind finding the nth Fibonacci number in O(log n) time using matrix exponentiation.
Then, write a highly optimized Python function to implement it. Provide your reasoning inside a <think> block."""
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=1024,
temperature=0.6,
do_sample=True
)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=False))