Ryze-125M
A 125M parameter GPT-2 style language model trained from scratch.
Model Details
| Property | Value |
|---|---|
| Architecture | GPT-2 (decoder-only transformer) |
| Parameters | ~125M |
| Layers | 12 |
| Attention Heads | 12 |
| Hidden Dimension | 768 |
| Context Length | 512 tokens |
| Vocabulary Size | 50,000 |
| Training Steps | 23,000 |
| Tokens Seen | 376,832,000 (~377M) |
| Best Val Loss | 4.3596 |
Training Data
- Primary: HuggingFaceFW/fineweb (sample-10BT)
- Validation: HuggingFaceFW/dclm_100BT
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Norman89107/ryze-125m")
tokenizer = AutoTokenizer.from_pretrained("Norman89107/ryze-125m")
input_text = "The future of artificial intelligence"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Configuration
- Micro Batch Size: 8
- Gradient Accumulation Steps: 4
- Effective Batch Size: 32 sequences (16,384 tokens)
- Learning Rate: 0.0003 (with ReduceLROnPlateau scheduler)
- Weight Decay: 0.1
- Max Gradient Norm: 1.0
- Dropout: 0.1
- Bias: Disabled
Custom Tokenizer
This model uses a custom BPE tokenizer trained on 100,000 documents with a vocabulary of 50,000 tokens.