Open-MOPD-SmolLM3-3B-MixSFT
This model is the mixed-domain supervised fine-tuning initialization used by the Open-MOPD pipeline. All three domain-specific RL teachers and the final multi-teacher distilled student start from this checkpoint.
The model is derived from HuggingFaceTB/SmolLM3-3B-Base through four epochs
of supervised fine-tuning on math, code, and instruction-following data. The
training setup uses global batch size 128, learning rate 4e-5, cosine decay
with 3% warmup, a maximum sequence length of 32,768, and 30,116 optimization
steps.
Domains are balanced by response-token count rather than example count. This prevents the 820K shorter instruction-following responses from overwhelming the smaller but longer math and code corpora. After balancing, math, code, and instruction following contribute approximately 37.3%, 28.1%, and 34.6% of training response tokens.
Results
| AIME24 | AIME25 | Math | LCBv5 | LCBv6 | Code | IFEval | IFBench_test | IF | Overall |
|---|---|---|---|---|---|---|---|---|---|
| 15.63 | 20.26 | 17.95 | 15.99 | 19.20 | 17.60 | 66.91 | 16.00 | 41.46 | 25.67 |
Evaluation protocol
- Math: AIME24 and AIME25, avg@64, temperature 0.6.
- Code: LiveCodeBench v5 and v6, avg@10, temperature 1.0.
- Instruction following: IFEval and IFBench_test,
n=1, temperature 1.0, withenable_thinking=true.
All evaluations use max_model_len=32768, top_p=0.95, top_k=-1, and
stop_token_ids=[128012]. Scores are averaged per dataset, then per domain,
followed by a macro-average across domains.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "BytedTsinghua-SIA/Open-MOPD-SmolLM3-3B-MixSFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype="bfloat16",
device_map="auto",
)
messages = [{"role": "user", "content": "Write a Python function that merges two sorted lists."}]
inputs = tokenizer.apply_chat_template(
messages,
return_tensors="pt",
add_generation_prompt=True,
enable_thinking=True,
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=4096, temperature=0.6, top_p=0.95, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
Role in the pipeline
SmolLM3-3B-Base -> MixSFT initialization -> Math/Code/IF RL teachers ->
Open-MOPD Final.
Model specifications
- Architecture:
SmolLM3ForCausalLM - Parameters: approximately 3B
- Layers: 36
- Vocabulary size: 128,256
- Weights: BF16, approximately 6.2 GB
- Includes tokenizer and chat template
The published tokenizer metadata is compatible with Transformers 4.x and produces the same tokenization and chat-template sequences as the original training artifact.