Qwen3-4B OpenThoughts3 Math SFT — Step 500
This is the Hugging Face export of global step 500 from an experimental
supervised fine-tuning run of Qwen/Qwen3-4B on the math portion of
open-thoughts/OpenThoughts3-1.2M.
The training data contains 103,760 examples built from OpenThoughts3 math
traces. Only complete responses with a closed reasoning block and a boxed final
answer were retained. The open target format was used: the Qwen3 thinking
block is left empty and the source reasoning plus final answer are trained in
the visible response channel.
Training configuration
- Checkpoint:
global_step_500(the run was stopped after this checkpoint) - Objective: full-parameter SFT
- Global batch size: 256
- Micro batch size per GPU: 1 with dynamic batching
- Maximum sequence length: 32,768 tokens
- Optimizer: AdamW
- Learning rate: 5e-6
- Scheduler: cosine, 3% warmup
- Weight decay: 0.01
- Gradient clipping: 1.0
- Precision: bfloat16
- Hardware for this run: 4 GPUs with sequence parallel size 4
Intended use
This checkpoint is intended for research on mathematical reasoning, SFT, and on-policy distillation. It is an intermediate experimental checkpoint, not a production model. AIME evaluation for this checkpoint has not yet been added to this model card.
Inference
Use the tokenizer and chat template shipped in this repository. For the
non-thinking behavior used during training, render prompts with
enable_thinking=False when supported by your inference stack.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "seomh/Qwen3-4B-OpenThoughts3-Math-SFT-step500"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype="auto",
device_map="auto",
)
Data note
Filtering incomplete OpenThoughts3 generations changes the difficulty distribution: questions whose sampled solutions repeatedly hit the source generation limit are underrepresented. Results should therefore be interpreted as training on the complete-answer subset rather than the full unfiltered math distribution.