Claudius Purus 110M
A generative language model with a knowledge cutoff of 0 BCE. Trained from scratch — architecture, tokenizer, and weights — exclusively on text composed before the Common Era. It has never seen a word written after antiquity. Ask it who rules, and the only possible answer is ancient.
To our knowledge (verified against HF, arXiv, and the vintage-LLM community as of mid-2026), this is the first generative model with a genuine pre-CE training cutoff; the earliest prior from-scratch cutoff was 1875 CE.
The five facts
- 25.85M training tokens — the entire defensibly-pre-CE, license-clean corpus we could assemble: Ancient Greek, Latin, Akkadian, Sumerian, Biblical Hebrew (consonantal), and Classical Chinese. 953 works spanning ~21 centuries of composition (22nd c. BCE – 1st c. BCE).
- Every source dated and auditable: inclusion is governed by a public dating manifest (per-author dates, confidence, and rationale — including the exclusions: Strabo, Ovid, the Periochae, pseudo-guwen chapters…). "Pre-CE" is a config file, not a vibe.
- 110.9M parameters, GPT-2-small architecture (12L/768/12H, 1024 ctx, 32k byte-level BPE trained on this corpus only), fp16, 16 epochs, seed 1453. Exported with exact (0.00e+00) logit parity to HF format.
- Best val loss 3.783 (ppl 44.0) on held-out ancient works (by-work split). Per-language numbers below — read the caveat first.
- Total training cost: about $2 of rented H100 time. The corpus, not compute, is the binding constraint: there is no more pre-CE text to scale onto.
What it is — and is not
It is a rhapsode, not an oracle: BabyLM-scale results apply — strong grammatical and stylistic competence, near-zero world-model. It writes period-true Ciceronian prose, hexameter-shaped Greek with genuine Homeric formulae, psalm-register consonantal Hebrew, annalistic Classical Chinese, and cuneiform-transliteration Akkadian — and it hallucinates freely within those registers. It answers questions the way a base model does: by continuing text. (Short Latin questions tend to summon Roman comedy; short Greek questions summon Platonic dialogue. This is a feature: the model is a detector for the register-distribution of surviving ancient literature.)
Known biases, stated plainly: the corpus is what antiquity's elites wrote and later ages preserved — literate, male, canonized. The model simulates surviving literature, not ancient populations. No benchmark can validate "thinks like 50 BCE"; our evals measure linguistic fidelity only. Hebrew is the unpointed consonantal text (the vowel points are medieval additions).
Evaluation
| lang | val tokens | NLL/token | per-token ppl |
|---|---|---|---|
| Akkadian | 110,579 | 2.537 | 12.6 |
| Ancient Greek | 88,155 | 3.781 | 43.9 |
| Latin | 36,470 | 4.739 | 114.3 |
| Classical Chinese | 45,397 | 6.209 | 497.2 |
| ALL | 280,601 | 3.808 | 45.1 |
⚠ Per-token perplexity is not comparable across languages: Akkadian transliteration splinters into ~4.4 predictable syllable-tokens per word, while each Chinese character is one whole-morpheme token. Per-word NLL inverts the ranking. Both views ship in the eval notes. The validation split contains no Hebrew or Sumerian works (hash-split coverage gap, noted for v2).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("notyost/claudius-purus-110m")
model = AutoModelForCausalLM.from_pretrained("notyost/claudius-purus-110m")
ids = tok("Non ad usum. Non ad probandum. Ludendi causa.",
return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True,
temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
Few-shot works and is period-appropriate: prime it with a master–pupil exchange (Platonic dialogue, catechism form) and it will hold the format for a beat or two. Expect drift; enjoy the drift.
Family
- claudius-purus-110m (this model): 6 languages, all training sources CC BY-SA / CC0 / public domain → released CC BY-SA 4.0.
- claudius-purus-110m-g: adds Vedic Sanskrit (GRETIL) and the Epic of Gilgamesh (eBL) — both CC BY-NC-SA sources — and is therefore released CC BY-NC-SA 4.0, honoring the licensors' one request. Research and play: use -g. Anything commercial: use this one.
Provenance & thanks
See ATTRIBUTION.md. Built on the shoulders of Perseus, Open Greek & Latin, ORACC, Tanach.us, and Kanripo — decades of scholarly digitization made a model like this possible for the price of a sandwich. First words, spoken at release through this very export, given its own founding incantation ("Not for use. Not for proof. For the joy of the making."):
Quid faciam? … Nunc ad omnia. Ridiculum exoritur: nam hic in te est. "What shall I do? … Now — to everything. The ridiculous arises: for it is here, in you."