Small-1B DPO Model
DPO (Direct Preference Optimization) model — 1B Transformer++ base, post-trained on top of SFT (kenpeter123/small-1b-sft) using preference pairs. Reference model = policy at init (standard DPO from the SFT anchor). 8-bit Adam, bf16, stage-isolated checkpoint.
| Spec | Value |
|---|---|
| Params | 1,031,898,624 (~1.03B) |
| Hidden | 1536 · 32 layers · 12 attn heads · 4 KV (GQA) · head_dim 128 |
| FFN | 4608 SwiGLU |
| Seq len | 2048 (train) / 8192 (config) |
| Vocab | 49152 (SmolLM2-135M tokenizer, BPE) |
| DPO | steps 300 · lr 1e-05 · β 0.1 · sigmoid loss |
| Optimizer | bitsandbytes Adam8bit, bf16 |
Preference pairs: raw problem prompt (chosen = gold correct solution, rejected = SFT model's
wrong/echo generation). Raw training checkpoint included as dpo_best.pt
(model_state_dict only, bf16).