RoBERTa dLLM · WikiText-2
This is a research checkpoint that turns roberta-base, a BERT-style bidirectional
masked-language model, into a masked discrete language diffusion model (dLLM).
It is the checkpoint served by
Caffin/bert-dllm.
Training
- Base model:
roberta-base - Dataset:
Salesforce/wikitext,wikitext-2-raw-v1, training split - Canvas length: 256 tokens
- Fixed conditioning prefix: first 16 tokens are never masked
- Noise schedule: one probability sampled per batch from
1.0, 0.9, ..., 0.1 - Loss: original token IDs only at positions selected for masking
- Run: 1,000 steps, batch size 4, learning rate
5e-5, seed 42 - Hardware: local Apple Silicon MPS
Sequences with no token after the fixed prefix are excluded because they contain no position that can contribute to the diffusion loss.
Inference
Start from a 256-token canvas containing the preserved prefix and 240 mask tokens. For ten reverse passes, predict every masked position in parallel, sample replacement tokens, and re-mask 90% down to 0% of the generated canvas. The Space exposes both:
- Article reproduction: uniform random re-masking.
- Confidence-aware: retain the most confident positions before the following pass.
The latter is included as a decoding comparison; it is not presented as the exact reproduction mode.
Limitations
This is a short WikiText-2 research fine-tune designed to demonstrate the masked diffusion objective and reverse process. It is not instruction-tuned, may repeat text or produce weak continuations, and should not be treated as a production language model.