Claudius 400M: a Latin-only language model
A 342M-parameter language model pretrained from scratch on Latin only: about 6.8 billion tokens of classical, medieval and neo-Latin text, trained compute-optimally (Chinchilla, about 20 tokens per parameter), then annealed on classical Latin and Latin Wikipedia so that it writes clean classical and expository Latin. Its continuations are Latin and nothing else: it has never seen English.
This is the recommended model. The pretrained model before the anneal is fbandov/claudius-latin-400m-base.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("fbandov/claudius-latin-400m")
model = AutoModelForCausalLM.from_pretrained("fbandov/claudius-latin-400m")
ids = tok("Gallia est omnis divisa in partes tres", return_tensors="pt").input_ids
out = model.generate(ids)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
- Any Latin input works. The tokenizer normalises to the classical orthography by itself: accents and macrons are dropped, letters are uppercased, V becomes U, J becomes I. "Salvē, amīce" and "SALUE, AMICE" are the same input.
- Output is in that orthography: capitals, U for both U and V. For display, write U as V (SALVE, VERBVM); on this text that is a one-to-one swap.
- Generation defaults are set (sampling, temperature 0.8, top-p 0.95, repetition penalty 1.2, no repeated 4-grams, 80 new tokens). Without the repetition controls, 7.4% of continuations fell into a loop; with them, 0%.
Training
| architecture | Llama: 24 layers, hidden 1024, 16 heads, MLP 2816, tied embeddings |
| parameters | 342M |
| context | 2,048 tokens |
| pretraining | 6.84B tokens (about 20 per parameter), AdamW (beta2 0.95, weight decay 0.1), peak lr 3e-4, cosine schedule to 10%, about 0.5M tokens per step |
| anneal | 136 steps at lr 3e-5 over 33.5M tokens (2 epochs): the classical core (Latin Library and Perseus) plus Vicipaedia |
| anneal data cleaning | edition line and section numbers removed; sentences that carry numbers as content dropped whole, not left with gaps |
| tokenizer | byte-level BPE, 32,768 tokens, trained on the normalised corpus, with the normaliser built in |
Training data
Latin only. No English or other modern language, not even translations.
| source | what it is | share of pretraining text | license |
|---|---|---|---|
| latin-literature-dataset-170M | classical, medieval and neo-Latin (Corpus Corporum) | 95% | not stated by the dataset |
| The Latin Library (via CLTK) and Perseus canonical-latinLit | classical prose and verse | 4% | CC BY-SA / public domain per text |
| Vicipaedia (Latin Wikipedia) | modern expository Latin | 1% | CC BY-SA 4.0 |
The text was filtered for Latin (a language score, with rival languages rejected), normalised, and deduplicated globally (exact paragraph hashing plus MinHash-LSH). Because it includes CC BY-SA text, the weights are released under CC BY-SA 4.0.
Evaluation
Bits per byte on a whole-author holdout. No author in the evaluation set appears in training. The floor, an interpolated character bigram, was fixed before any model was trained.
| model | params | tokens | bits per byte |
|---|---|---|---|
| character-bigram floor | 3.518 | ||
| Claudius 150M | 110M | 2.2B | 1.351 |
| Claudius 400M base | 342M | 6.84B | 1.091 |
| Claudius 400M | 342M | 6.84B + 67M anneal | 1.097 |
Output quality. Measured per 1,000 words on continuations of 40 fixed prompts (classical, everyday, modern), 3 samples each:
| edition numbers | church vocabulary | fused words | loops | |
|---|---|---|---|---|
| 400M base | 10.9 | 17.0 | 0.4 | 0% |
| 400M | 0.0 | 11.6 | 0.0 | 0% |
"Church vocabulary" counts words such as DEUS, DOMINUS, ECCLESIA, SANCTUS. The classical core of the corpus has 10.8 per 1,000 and the medieval bulk 28.5.
Memorisation. No 13-word run of generated text occurs verbatim in the training text, for either model.
Limitations
- It invents things. It writes fluent, grammatical Latin, but at 342M parameters it has no reliable knowledge. Asked about Gaul, it will name islands and peoples that do not exist. Use it to generate Latin, not facts.
- It continues text, it does not chat. This is a base model with no instruction or dialogue training. Prompt it with the start of the text you want.
- A medieval accent. 95% of the pretraining text is medieval or later. The anneal pulls the register towards classical and expository Latin, but vocabulary and style still drift post-classical, and religious themes come up more often than in Cicero.
- Book structure. It sometimes writes chapter and book headings (CAPUT II, LIBER UI), as the source editions do.
- No Greek, no lowercase in output. Greek was removed from the training text, and output is always in capitals.
License
CC BY-SA 4.0. Credit "Claudius (fbandov)" and share derivatives under the same license.