Causal Byte LM — Shakespeare
Custom PyTorch decoder trained from random initialization on Tiny Shakespeare. 635,776 parameters, 128-byte context, three layers, tied embeddings, RMSNorm, SwiGLU and RoPE. This is a research artifact, not an instruction model or a general assistant.
Evaluation
| Model | Parameters | Steps | Test bits/byte ↓ | Byte perplexity ↓ |
|---|---|---|---|---|
| rope-seed42 | 635,776 | 3,000 | 2.4870 | 5.6062 |
| no-rope-seed42 | 635,776 | 3,000 | 2.8659 | 7.2898 |
| bigram, add-0.1 | 65,536 | 0 | 3.6243 | 12.3315 |
One seed (42), matched 3000-step neural training budgets. Validation-selected checkpoint; 111,488 separate test-byte targets. Byte perplexity differs from word/token perplexity. Reproduction, ablation curves and causal-mask tests: https://github.com/zoga228/causal-lm-lab.
Load
Clone the source repository and install its requirements. model.py in that repository is the audited custom implementation.
from huggingface_hub import snapshot_download
from model import ByteLM
path = snapshot_download("Zangeeeeerange/causal-byte-lm-shakespeare",
allow_patterns=["model.safetensors", "config.json", "tokenizer.json"])
model = ByteLM.from_pretrained(path)
print(model.generate("ROMEO:\n",new_bytes=200,seed=42))
This repository does not support Transformers AutoModel, the default Hugging Face inference widget or hosted inference. Checkpoints use SafeTensors; no pickle is required. Encoding is UTF-8 bytes; generated bytes decode with replacement for invalid UTF-8 sequences.
Training data and limitations
Public-domain Shakespeare excerpts from https://github.com/karpathy/char-rnn; source revision and checksum are in the GitHub data manifest. Contiguous 80/10/10 byte splits precede all windowing. 892,315 unique training bytes, 12,288,000 sampled target-byte presentations. Repetition is not additional unique data. Models use a single seed and a single corpus; no cross-document semantic deduplication claim is made.
Generated text often contains invented words, repetitions and weak coherence. No factuality, safety, instruction-following or multilingual evaluation was performed. Do not use the model for factual or consequential decisions. AI tools assisted development; metrics come from executed experiments. Code and weights are MIT; corpus text is public domain.