Qwen3-4B long-CoT SFT
A vanilla supervised fine-tune of Qwen/Qwen3-4B-Base on long chain-of-thought math data.
No architectural changes — a standard Qwen3 model, loadable with from_pretrained.
We release it as a full-context reference baseline: a plain SFT of this base model at this data scale, useful as a control to measure other fine-tunes against, and as a frozen teacher for distillation.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "jackcai1206/qwen3-4b-longcot-sft-vanilla"
model = AutoModelForCausalLM.from_pretrained(REPO, dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained(REPO)
Precision
Training kept fp32 master weights (16.1 GB). Downstream use constructs the model in bf16, so those weights are
truncated to bf16 on load — this export is bf16 and therefore bit-identical to what training and inference
actually consume, at half the size. Verified tensor by tensor against the fp32 checkpoint. lm_head is tied to
embed_tokens in the source checkpoint (same storage), so it is reconstructed by from_pretrained rather than
stored separately.
Evaluation
MATH-500 (strict): 87.4.