diffusion-lm-nano
This card describes the measured checkpoint from release b38dbab. The model
is a 44.75M-parameter bidirectional Transformer trained to predict masked
tokens. It is preserved as a collapse and stopping-study result, not as a
usable language model.
Result
The 10,000-step checkpoint has held-out masked-token accuracy 0.36029 and
cross entropy 5.14059. The majority <|EOW|> token occupies 0.36205 of the
same complete held-out sequences. The saved 256-token unconditional sample
contains only <|EOW|> and decodes to spaces.
Adaptive calibration tested 12 settings on 32 sequences and evaluated the selected setting on a separate 128-sequence window. It selected entropy threshold zero and confidence patience zero. Both stopping rules were disabled. Every held-out example used all 32 passes.
The recorded timing comparison measured 0.25060 seconds for the diffusion
path and 1.40847 seconds for the autoregressive path. The 5.62x ratio is
limited to batch size one, 256 output tokens, and the exact recorded execution
paths. Output quality was not matched and the diffusion sample was blank.
Provenance
The checkpoint SHA-256 is
96f78f1d51fab2a365142bf2a861feb109633a189658fd97864087e913c3a9db.
It records source commit 2bb34f0516d531acfbcd5df8ee66c5e8cd80d1b6
and a clean Git state at training start.
The checkpoint does not record the training hardware or PyTorch runtime. Later
held-out evaluation, adaptive decoding, and timing ran on one NVIDIA RTX A6000
with PyTorch 2.4.1+cu124.
The input shard and tokenizer come from kotlarmilos/gpt2-nano at revision
ee547b4112bce2d2557dd0975038035bab6c86eb. Exact hashes are in
artifacts/manifest.json.
Intended use
Use this checkpoint to inspect majority-token collapse, reproduce the held-out controls, and study the adaptive stopping implementation. Do not use it for text generation, factual tasks, or deployment.
Only load checkpoint files from trusted sources. PyTorch checkpoint files can contain unsafe serialized data.
The implementation is available at
https://github.com/kotlarmilos/diffusion-lm-nano.