a2d-gpt-neox-160M
Base autoregressive → MLM diffusion models derived from the EleutherAI Pythia family.
These are base models only (no supervised fine-tuning / instruction tuning).
Model description
- Model type: Decoder-only Transformer converted to Masked Language Modeling (MLM) diffusion architecture
- Base architecture: Pythia (GPT-NeoX)
Variants
| Size | HF Base | Checkpoint | Total Params | Non-embedding Params | Tokens seen (pretrain) | Tokens / Param |
|---|---|---|---|---|---|---|
| 70M | EleutherAI/pythia-70m | step 512 | 70.4M | 18.9M | ~1.07B | ~15 + continued |
| 160M | EleutherAI/pythia-160m | step 1000 | ~162M | ~125M | ~2.1B | ~13 + continued |
| 410M | EleutherAI/pythia-410m | step 3000 | 405.3M | ~350M | ~6.29B | ~15.5 + continued |
Original Pythia pretraining & checkpoint selection (EleutherAI):
All models were initialized from Pythia checkpoints selected according to Chinchilla scaling laws (Hoffmann et al., 2022) targeting ~15 tokens per parameter (slightly undertrained regime) for consistency across the model family.
- Dataset: The Pile
- Total tokens in Pythia run: 299.9B
- Checkpoint interval: every 2,097,152 tokens (~2M batch size)
Selected checkpoints:
- 70M: step 512 (~1.07B tokens, ~15 tok/param)
- 160M: step 1000 (~2.1B tokens, ~13 tok/param)
- 410M: step 3000 (~6.29B tokens, ~15.5 tok/param)
Intended uses & limitations
- Research on autoregressive-to-diffusion conversion
- Continued pre-training and domain adaptation experiments
- Analysis of scaling behavior in hybrid MLM diffusion models
- Not intended for direct deployment or chat/instruction use (base models only)
Training and evaluation data
Continued pre-training: All models were further trained on the same subset of ~3B tokens sampled from FineWeb-Edu using (masked) causal language modeling objective.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 64
- eval_batch_size: 64
- seed: 46
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.85,0.94) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 468
- training_steps: 4687
How to Get Started with the Model
Use the following code to get started with the model:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "avgJo3/a2d-gpt-neox-160M" # or 70m / 160m variant
revision = "main" # or specific checkpoint tag if uploaded separately
model = AutoModelForCausalLM.from_pretrained(
model_name,
revision=revision,
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
model_name,
revision=revision,
trust_remote_code=True
)
Framework versions
- Transformers 5.0.0
- Pytorch 2.10.0+cu128
- Datasets 4.8.5
- Tokenizers 0.22.2
Citation
@article{biderman2023pythia,
title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling},
author={Biderman, Stella and others},
journal={ICML},
year={2023}
}