KainoBERTa
KainoBERTa is a roberta-base masked language model for Ancient Greek, trained
from random initialization on the second derivative of the
Urdatorn/AncientGreek-no-sphragis
corpus (revision 73f02698199c404d85acdc875b0f73d0741caa10). It does not
inherit the weights of FacebookAI/roberta-base; only the architecture is
reproduced. It is the architecture control of
Urdatorn/KainoBERT-sphragis
on the third edition of the
Sphragis leaderboard.
Why this model replaces the first KainoBERTa
This repository first held a KainoBERTa pretrained on a corpus from which only the exact lines of the benchmark had been removed, so it had read the benchmark's works in other editions. Its memorisation probe showed no gap, but at that model's accuracy the probe has little power, and the leaderboard's standard, applied to KainoBERT on 2026-09-24, is that no published encoder may have read the benchmark's works at all. That model was withdrawn on 2026-09-25 and its weights deleted.
The architecture control
KainoBERT and KainoBERTa read the very same prepared corpus: the same
32,768-token byte-level BPE, the same packed 1,024-token blocks (80% with
polytonic diacritics, a deterministic 20% through grc_utils.only_bases),
the same 30% dynamic masking, the same fused AdamW, effective batch of 512
sequences, weight decay and stopping rule. They differ in the architecture
alone, ModernBERT-base against roberta-base, and in one training-length
setting forced by the architecture: KainoBERTa's high-learning-rate phase is
twelve epochs where KainoBERT's is four, because a four-epoch phase left the
roberta architecture on its initial plateau (validation loss 6.54; that run's
metrics are kept beside the training code). Everything after the plateau is
the shared recipe.
Training
- Architecture: RoBERTa base, 12 layers, hidden size 768, 12 heads, 111,634,688 parameters; no inherited weights
- Phase one: 12 epochs at peak learning rate 3e-4 with 5% warmup, decaying to 3e-5: 7,584 steps, 3.98B tokens presented, validation loss 3.082
- Phase two (continuation from the phase-one weights): learning rate 3e-5, no warmup, stopped by early stopping after six evaluations without a 0.002 improvement in validation loss: 21,750 further steps, 11.40B tokens presented, best validation loss 2.731 (perplexity 15.4)
- Total tokens presented: 15.38B
The corpus removes every benchmark work by identifier and by text, every record holding a six-word window of a benchmark unit, and every exact line; a rescan of the finished corpus finds zero benchmark windows and zero matching lines. The masked-token memorisation probe on the leaderboard reports where this model places the benchmark's test sentences between the corpus text it saw and the text it did not.
Full provenance, hyperparameters and the loss history of the second phase
are in training_metrics.json. The training code is
Urdatorn/sphragis_models
(slurm/train_kainoberta_v2.slurm, slurm/continue_kainoberta_v2.slurm,
sphragis_models/kainobert_pretrain.py --architecture roberta).
Licence
The corpus is a mixed-licence compilation. Consult each source record's
license field before redistribution or downstream use.
Limitations
The exclusion removes benchmark works, six-word windows of benchmark units and exact lines. It cannot rule out paraphrases, quotations shorter than six words, or scholia and commentaries that embed benchmark text in other words. It also removed the benchmark authors' works wholesale, so the model has read less of exactly the authors the benchmark asks about.