LLaDA-8B full-sequence SFT on mix60k
This model is the full-sequence SFT control model described in the paper Register Tokens for Bounded-State Reasoning in Diffusion Language Models. Matched full-sequence-SFT control for the dLLM Registers paper. This model has no register or discrete-text carry channel.
Training
- Initialization:
GSAI-ML/LLaDA-8B-Base - Data: the exact 60K mix of OpenMathInstruct-2 and OpenCodeInstruct in
albertge/mix60k-math-code-sft - Four independently noised diffusion-loss optimizer passes per example
- Natural/dynamic sequence lengths, capped at 1024 completion tokens
- One epoch, batch size 1 per rank, learning rate 2e-5, weight decay 0.1
- Exact runner:
SFT/sft_train_vanilla.pyandmodal_jobs/vanilla_sft.py
Recorded evaluation
| Protocol | GSM8K | MATH500 |
|---|---|---|
| One 1024-token canvas | 63.46 (837/1319) | 21.8 (109/500) |
| Fresh 8 x 128, no carried state | 56.33 (743/1319) | 23.8 (119/500) |
For math, evaluation scores the first emitted answer. Under fresh 8 x 128, later chunks cannot read earlier generated text and carry no hidden state.
Repository: https://github.com/SprocketLab/dllm-registers