KainoBERT
KainoBERT is a ModernBERT-base masked language model for Ancient Greek,
trained from random initialization on the second derivative of the
Urdatorn/AncientGreek-no-sphragis
corpus (revision 73f02698199c404d85acdc875b0f73d0741caa10). It is the
encoder of the third edition of the
Sphragis leaderboard:
every work, passage and line of the Sphragis and Sphragis-Metre benchmarks
was removed from its pretraining corpus, so that its authorship scores cannot
come from having read the test sentences during pretraining.
Why this model replaces the first KainoBERT
This repository first held a KainoBERT pretrained on a corpus from which only the exact lines of the benchmark had been removed. That model was withdrawn on 2026-09-24 and its weights deleted. A masked-token memorisation probe (masked-token accuracy on the benchmark's test sentences against held-back corpus text the model saw and text it did not) places that model's accuracy on the test sentences at its seen level: it had read them in other editions of the same works.
The present KainoBERT is pretrained on a corpus built to make that impossible:
- works by identifier: 67 works and 20 whole authors of the benchmarks leave the OGA source by their TLG identifiers;
- works by text: 628 further works, from any source, leave the corpus because at least three benchmark units occur in them as six-word windows;
- records by window: any remaining record holding a six-word window of a benchmark unit leaves every source (55,222 records in the pristine tier, 243,925 in the repaired tier);
- lines: the first edition's exact normalised-line pass runs on what is left.
A rescan of the finished corpus finds zero benchmark windows and zero matching lines. The same probe run on this model places its accuracy on the test sentences below its unseen level (0.329 against 0.388 unseen and 0.380 seen), where the withdrawn model sat at 0.460 against 0.434 unseen and 0.463 seen.
The price of the cleaning is a weaker language model: the corpus lost the benchmark authors' works wholesale, and this model reaches a validation loss of 2.38 (perplexity 10.8) where the withdrawn one reached 2.19. On the leaderboard its rows sat three to four macro-F1 points under the withdrawn model's on the in-work sentence track and within a few points on the work-held-out tracks: the memorised text was worth a few points, not the ranking.
Corpus and normalization
All text is lowercased and depunctuated. 80% of records retain polytonic
diacritics; a deterministic 20% are transformed with grc_utils.only_bases
while preserving word boundaries. A fresh 32,768-token byte-level BPE
vocabulary was trained on the cleaned corpus.
Training
- Architecture: ModernBERT base, 22 layers, hidden size 768, 12 heads,
136,120,832 parameters; no weights inherited from
answerdotai/ModernBERT-base - Blocks of 1,024 tokens, effective batch 512 sequences
- Objective: masked language modelling, 30% dynamic training masks, 15% validation masks
- Phase one: 4 epochs, fused AdamW, peak learning rate 3e-4 with 5% warmup, 2,528 steps, 1.33B tokens presented, validation loss 2.706
- Phase two (continuation from the phase-one weights): learning rate 3e-5, no warmup, stopped by early stopping after six evaluations without a 0.002 improvement in validation loss: 11,500 further steps, 6.03B tokens presented, best validation loss 2.380 (perplexity 10.8)
- Total tokens presented: 7.35B
Full provenance, hyperparameters and the loss history of the second phase
are in training_metrics.json. The training code is
Urdatorn/sphragis_models
(slurm/train_kainobert_v2.slurm, slurm/continue_kainobert_v2.slurm,
sphragis_models/work_window_excluded_corpus.py).
Licence
The corpus is a mixed-licence compilation. Consult each source record's
license field before redistribution or downstream use.
Limitations
The exclusion removes benchmark works, six-word windows of benchmark units and exact lines. It cannot rule out paraphrases, quotations shorter than six words, or scholia and commentaries that embed benchmark text in other words. It also removed the benchmark authors' works wholesale, so the model has read less of exactly the authors the benchmark asks about.