KainoBERT
KainoBERT is a ModernBERT-base masked language model for Ancient Greek, trained
from random initialization with a fresh 32,768-token byte-level BPE vocabulary
(22 layers, hidden size 768, 12 heads, 136M parameters, 1,024-token context).
It does not inherit the English weights of answerdotai/ModernBERT-base;
only the architecture is reproduced.
This is the un-audited, most complete version of KainoBERT, for use in the
wild. It has read the whole of
Ericu950/AncientGreek,
including every work in the Sphragis and Sphragis-Metre authorship benchmarks.
It must not be evaluated on those benchmarks. For controlled,
contamination-free experiments use
Urdatorn/KainoBERT-sphragis,
which is this model's parent and has never seen a benchmark work, passage or line.
How it was trained
Three phases, all with 30 % dynamic masking, fused AdamW, bf16, an effective batch of 512 sequences of 1,024 tokens, on two GH200 GPUs.
- Pretraining on
Urdatorn/AncientGreek-no-sphragis(second derivative, revisionv2-sphragis-1e6d8b58d956e84aec7c1c778bef036bd0286fa9): AncientGreek with every benchmark work, every other edition or commentary of one, every record carrying a benchmark window and every benchmark line removed. 323,225 training blocks (331M tokens), peak learning rate 3e-4, cosine schedule, four epochs. - Continuation on the same corpus from the terminal learning rate of
phase one (3e-5, fresh optimizer), stopped after six evaluations without a
0.002 improvement in validation loss (epoch 18.2, loss 2.380). The result is
Urdatorn/KainoBERT-sphragis. - Continuation on the removed material (this model): everything phase
one's corpus had excluded, reconstructed by diffing AncientGreek
(revision
6ac90787c669a7e9218d6d4675a029fa3f10ed99) against the second derivative: 313,840 records dropped whole (750M characters, both quality tiers) plus the 4,092 removed lines of records that were kept. Packed with the same tokenizer and surface rules (lowercased, depunctuated, 80 % polytonic, 20 % bare letters) into 161,745 blocks (166M tokens), 0.2 % held out for validation. Same recipe as phase two: learning rate 3e-5, no warmup, cosine to 10 %, early-stopping rule 6 × 0.002, ceiling 30 epochs. Run in legs of 30 epochs, each restarting the cosine schedule from 3e-5 with a fresh optimizer, as phase two did after phase one. Validation loss on the removed material: 2.614 for the phase-two model (unseen text); 1.977 after leg 1; 1.806 after leg 2; 1.706 after leg 3; 1.639 after leg 4 (this revision). Per-leg gains 0.637, 0.171, 0.100, 0.067 halve roughly each leg. A leg is added while a fresh schedule still buys more than the stopping rule's margin; the Hub revision history keeps every leg.
Full hyperparameters, the packed-corpus manifest and the loss history are in
training_metrics.json. The pipeline is in
Urdatorn/sphragis_models
(scripts/build_removed_corpus.py, scripts/prepare_removed_corpus.py,
slurm/continue_kainobert_v3.slurm).
Input surface
Text should be lowercased and depunctuated, polytonic marks kept, as in
sphragis_models.kainobert_corpus.polytonic_surface; the model also reads
bare-letter text. The tokenizer adds [CLS] … [SEP].
Licence
The corpus is a mixed-licence compilation. Consult each source record's
license field and the upstream dataset cards before reuse.
Created by Albin Thörn Cleland, Lund University (ORCID 0009-0003-3731-4038).