Odysseus-2
A tiny GPT-2-style language model (~320M parameters) trained from scratch on a 500MB slice of English Wikipedia.
Model Details
- Architecture: GPT-2 (
GPT2LMHeadModel) - Parameters: ~320M
- Layers: 24
- Attention heads: 16
- Embedding dimension: 1024
- Context length: 512 tokens
- Vocabulary: 16,000 BPE tokens (trained from scratch)
- Training data: ~500MB slice of English Wikipedia
- License: MIT
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "dogman189/odysseus-2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "The history of"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
- Trained from scratch on a ~500MB slice of English Wikipedia
- BPE tokenizer trained separately (vocab 16,000)
- ~2 epochs of training
- See the TinyWiki repository for the full training pipeline
Evaluation
Evaluated on the held-out val split (512-token sequences):
- Eval loss: 3.6507
- Perplexity: 38.50
Limitations
Small model trained on a limited Wikipedia slice — output quality is limited and may contain inaccuracies or repetition. Not suitable for production use without further training or fine-tuning.