KainoBERT-v2
KainoBERT-v2 is a ModernBERT-base masked language model for Ancient Greek,
trained from random initialization on the second derivative of the
Urdatorn/AncientGreek-no-sphragis
corpus (revision 73f02698199c404d85acdc875b0f73d0741caa10). It is the
clean encoder of the third edition of the
Sphragis leaderboard:
every work, passage and line of the Sphragis and Sphragis-Metre benchmarks
was removed from its pretraining corpus, so that its authorship scores cannot
come from having read the test sentences during pretraining.
Why this model exists
The published Urdatorn/KainoBERT-sphragis
was pretrained on a corpus from which only the exact lines of the benchmark
had been removed. A masked-token memorisation probe (masked-token accuracy on
the benchmark's test sentences against held-back corpus text the model saw
and text it did not) places that model's accuracy on the test sentences at its
seen level: it had read them in other editions of the same works.
KainoBERT-v2 is pretrained on a corpus built to make that impossible:
- works by identifier: 67 works and 20 whole authors of the benchmarks leave the OGA source by their TLG identifiers;
- works by text: 628 further works, from any source, leave the corpus because at least three benchmark units occur in them as six-word windows;
- records by window: any remaining record holding a six-word window of a benchmark unit leaves every source (55,222 records in the pristine tier, 243,925 in the repaired tier);
- lines: the first edition's exact normalised-line pass runs on what is left.
A rescan of the finished corpus finds zero benchmark windows and zero matching lines. The same probe run on KainoBERT-v2 places its accuracy on the test sentences below its unseen level (0.329 against 0.388 unseen and 0.380 seen), where the published model sits at 0.460 against 0.434 unseen and 0.463 seen.
The price of the cleaning is a weaker language model: the corpus lost the benchmark authors' works wholesale, and KainoBERT-v2 reaches a validation loss of 2.38 (perplexity 10.8) against the published model's lower loss. On the leaderboard its rows sit three to four macro-F1 points under the published KainoBERT on the in-work sentence track and within a few points on the work-held-out tracks.
Corpus and normalization
All text is lowercased and depunctuated. 80% of records retain polytonic
diacritics; a deterministic 20% are transformed with grc_utils.only_bases
while preserving word boundaries. A fresh 32,768-token byte-level BPE
vocabulary was trained on the cleaned corpus.
Training
- Architecture: ModernBERT base, 22 layers, hidden size 768, 12 heads,
136,120,832 parameters; no weights inherited from
answerdotai/ModernBERT-base - Blocks of 1,024 tokens, effective batch 512 sequences
- Objective: masked language modelling, 30% dynamic training masks, 15% validation masks
- Phase one: 4 epochs, fused AdamW, peak learning rate 3e-4 with 5% warmup, 2,528 steps, 1.33B tokens presented, validation loss 2.706
- Phase two (continuation from the phase-one weights): learning rate 3e-5, no warmup, stopped by early stopping after six evaluations without a 0.002 improvement in validation loss: 11,500 further steps, 6.03B tokens presented, best validation loss 2.380 (perplexity 10.8)
- Total tokens presented: 7.35B
Full provenance, hyperparameters and the loss history of the second phase
are in training_metrics.json. The training code is
Urdatorn/sphragis_models
(slurm/train_kainobert_v2.slurm, slurm/continue_kainobert_v2.slurm,
sphragis_models/work_window_excluded_corpus.py).
Licence
The corpus is a mixed-licence compilation. Consult each source record's
license field before redistribution or downstream use.
Limitations
The exclusion removes benchmark works, six-word windows of benchmark units and exact lines. It cannot rule out paraphrases, quotations shorter than six words, or scholia and commentaries that embed benchmark text in other words. It also removed the benchmark authors' works wholesale, so the model has read less of exactly the authors the benchmark asks about.