Claudius 400M Base: a Latin-only language model
The pretrained checkpoint of Claudius 400M: 342M parameters, pretrained from scratch on Latin only, about 6.8 billion tokens, trained compute-optimally (Chinchilla). It has not been annealed, so its register is that of its data: mostly medieval, scholastic and ecclesiastical. It also prints edition line and section numbers it learned from the sources.
For most uses, take fbandov/claudius-latin-400m instead. This checkpoint is for continued pretraining, fine-tuning, and comparison.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("fbandov/claudius-latin-400m-base")
model = AutoModelForCausalLM.from_pretrained("fbandov/claudius-latin-400m-base")
ids = tok("Sed de his hactenus", return_tensors="pt").input_ids
out = model.generate(ids)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
- Any Latin input works. The tokenizer normalises to the classical orthography by itself: accents and macrons are dropped, letters are uppercased, V becomes U, J becomes I. "Salvē, amīce" and "SALUE, AMICE" are the same input.
- Output is in that orthography: capitals, U for both U and V. For display, write U as V (SALVE, VERBVM); on this text that is a one-to-one swap.
- Generation defaults are set (sampling, temperature 0.8, top-p 0.95, repetition penalty 1.2, no repeated 4-grams, 80 new tokens). Without the repetition controls, 7.4% of continuations fell into a loop; with them, 0%.
Training
| architecture | Llama: 24 layers, hidden 1024, 16 heads, MLP 2816, tied embeddings |
| parameters | 342M |
| context | 2,048 tokens |
| tokens | 6.84B (about 20 per parameter), 13,911 steps of about 0.5M tokens |
| optimiser | AdamW, beta2 0.95, weight decay 0.1, peak lr 3e-4, cosine schedule to 10% |
| tokenizer | byte-level BPE, 32,768 tokens, with the classical normaliser built in |
Training data
Latin only. No English or other modern language, not even translations.
| source | what it is | share of pretraining text | license |
|---|---|---|---|
| latin-literature-dataset-170M | classical, medieval and neo-Latin (Corpus Corporum) | 95% | not stated by the dataset |
| The Latin Library (via CLTK) and Perseus canonical-latinLit | classical prose and verse | 4% | CC BY-SA / public domain per text |
| Vicipaedia (Latin Wikipedia) | modern expository Latin | 1% | CC BY-SA 4.0 |
The text was filtered for Latin (a language score, with rival languages rejected), normalised, and deduplicated globally (exact paragraph hashing plus MinHash-LSH). Because it includes CC BY-SA text, the weights are released under CC BY-SA 4.0.
Evaluation
Bits per byte on a whole-author holdout. No author in the evaluation set appears in training. The floor, an interpolated character bigram, was fixed before any model was trained.
| model | params | tokens | bits per byte |
|---|---|---|---|
| character-bigram floor | 3.518 | ||
| Claudius 150M | 110M | 2.2B | 1.351 |
| Claudius 400M base | 342M | 6.84B | 1.091 |
| Claudius 400M | 342M | 6.84B + 67M anneal | 1.097 |
Output quality. Measured per 1,000 words on continuations of 40 fixed prompts (classical, everyday, modern), 3 samples each:
| edition numbers | church vocabulary | fused words | loops | |
|---|---|---|---|---|
| 400M base | 10.9 | 17.0 | 0.4 | 0% |
| 400M | 0.0 | 11.6 | 0.0 | 0% |
"Church vocabulary" counts words such as DEUS, DOMINUS, ECCLESIA, SANCTUS. The classical core of the corpus has 10.8 per 1,000 and the medieval bulk 28.5.
Memorisation. No 13-word run of generated text occurs verbatim in the training text, for either model.
Limitations
- It invents things. It writes fluent, grammatical Latin, but at 342M parameters it has no reliable knowledge. Asked about Gaul, it will name islands and peoples that do not exist. Use it to generate Latin, not facts.
- It continues text, it does not chat. This is a base model with no instruction or dialogue training. Prompt it with the start of the text you want.
- A medieval accent. 95% of the pretraining text is medieval or later. The anneal pulls the register towards classical and expository Latin, but vocabulary and style still drift post-classical, and religious themes come up more often than in Cicero.
- Book structure. It sometimes writes chapter and book headings (CAPUT II, LIBER UI), as the source editions do.
- No Greek, no lowercase in output. Greek was removed from the training text, and output is always in capitals.
License
CC BY-SA 4.0. Credit "Claudius (fbandov)" and share derivatives under the same license.