Claudius Purus 110M-G
A generative language model with a knowledge cutoff of 0 BCE — the full seven-language variant, including Vedic Sanskrit and the Epic of Gilgamesh. Trained from scratch (architecture, tokenizer, weights) exclusively on text composed before the Common Era. It has never seen a word written after antiquity.
Why NC: two sources — GRETIL's Vedic corpus and the electronic Babylonian Library's Gilgamesh — are CC BY-NC-SA. Whether trained weights inherit a text license is legally unsettled; we honor the licensors' one request regardless, so this model is research/play only. For commercial use, its sibling claudius-purus-110m (clean six-language variant, CC BY-SA 4.0) exists for exactly that purpose.
The facts
- 27.92M training tokens: Ancient Greek, Latin, Akkadian, Sumerian, consonantal Biblical Hebrew, Classical Chinese, Vedic Sanskrit — plus the Standard Babylonian and Old Babylonian Gilgamesh. 996 works spanning ~21 centuries of composition. Every inclusion governed by a public dating manifest (dates, confidence, rationale — and the exclusions: Strabo, Ovid, Livy's Periochae, Śaṅkara's commentary, Qianlong's poems…).
- 110.9M parameters, GPT-2-small architecture (12L/768/12H, 1024 ctx, 32k corpus-trained byte-level BPE), fp16, 16 epochs, seed 1453. Exported with exact (0.00e+00) logit parity to HF format.
- Best val loss 3.664 (ppl 39.0); per-language below. Training cost ~$2 of rented H100. The corpus, not compute, is the binding constraint — there is no more pre-CE text.
Showcase: the ibru experiment
This model is the "after" of a controlled intervention. Its predecessor was identical except for one text: the Epic of Gilgamesh (13k words — 0.05% of the corpus, but ~40% of the model's lifetime exposure to Akkadian ibru, "friend"). Same seed, frozen tokenizer, byte-identical validation set; Latin/Greek control probes held steady. Result: the Akkadian negation cadence re-aimed from contract law ("he shall not sue, he shall not hand over") to mortality ("his strength does not return"; the ghost; the weary; the ancient), and the bare cry ib-rī began answering with first-person memory ("in the palace, he seated me"). One epic bent the word "friend" from the courtroom toward the grave. That is what this model is for: measuring what surviving texts do to meanings.
What it is — and is not
A rhapsode, not an oracle: strong period style and grammar (Ciceronian prose, Homeric formulae, psalm-register Hebrew, Brāhmaṇa ritual prose, cuneiform conventions — including, charmingly, the lacuna brackets of damaged tablets), near-zero factual reliability, no reasoning. It continues text; it does not chat. Biases stated plainly: the corpus is elite, literate, male, and survivorship-filtered — this simulates surviving literature, not ancient populations. Hebrew is unpointed consonantal text. Sanskrit binds weakly (its mass is formulaic ritual prose; expect the altar to swallow your prompt).
Evaluation
| lang | val tokens | NLL/token | per-token ppl |
|---|---|---|---|
| Akkadian | 110,681 | 2.499 | 12.2 |
| Sanskrit | 38,341 | 3.602 | 36.7 |
| Ancient Greek | 89,124 | 3.765 | 43.2 |
| Latin | 36,951 | 4.729 | 113.2 |
| Classical Chinese | 45,905 | 6.056 | 426.7 |
| ALL | 321,002 | 3.748 | 42.4 |
⚠ Per-token perplexity is incomparable across tokenization densities (Akkadian ≈ 4.4 predictable syllable-tokens/word; Chinese ≈ 1 morpheme-token/char); per-word NLL inverts the ranking (Greek is the model's strongest language). Val split contains no Hebrew/Sumerian works (hash-split coverage gap, noted for v2).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("notyost/claudius-purus-110m-g")
model = AutoModelForCausalLM.from_pretrained("notyost/claudius-purus-110m-g")
ids = tok("ib-ri", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=50, do_sample=True,
temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
Provenance & thanks
See ATTRIBUTION.md — Perseus, Open Greek & Latin, ORACC, Tanach.us, Kanripo, GRETIL, and the electronic Babylonian Library. Decades of scholarly digitization, honored license and all, made this possible for the price of a sandwich. Ib-ri na-piš-ti — my friend, my life — the model said it before it ever read the tablet where it belongs.