PebbleGPT-320M
A 320M-parameter dense transformer pretrained from scratch on 10B tokens. Base model — it continues text, it does not follow instructions.
Architecture
- 24 layers, hidden size 1024, intermediate 2816
- Grouped Query Attention: 16 query heads, 4 KV heads (ratio 4)
- RoPE (theta 10000), SwiGLU, RMSNorm, tied embeddings
- 2048 context, SmolLM2 tokenizer (49,152 vocab)
Training data (10B tokens)
| Source | Share |
|---|---|
| FineWeb-Edu | 42.5% |
| DCLM-Edu (edu_int_score >= 3) | 42.5% |
| Python-Edu | 10% |
| FineMath-4+ | 5% |
Training
AdamW (beta 0.9/0.95, weight decay 0.1 excluding embeddings), WSD schedule with peak LR 5e-4 and 10% decay, 2000 warmup steps. 19,073 steps at 524,288 tokens per step. ~18 hours on a single H100 SXM at ~35% MFU.
Final training loss: 2.507 (perplexity ~12.3)
Results
| Benchmark | Score |
|---|---|
| HellaSwag (acc_norm) | 30.5 |
| PIQA | 55.5 |
| ARC-easy (acc_norm) | 37.7 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"strectelite/PebbleGPT-320M", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("strectelite/PebbleGPT-320M")
Note: no KV cache implemented yet, so generation is slow.