19c-ModernBERT-mini-B — Newspaper Gated v2 (full-corpus chunked)
Continued MLM pretraining of 19c-modernbert-mini-b on the quality-filtered newspaper corpus, v2: full-corpus chunked packing.
This supersedes
19c-modernbert-mini-b-news-gated.
The v1 run silently trained on ~16% of the corpus — truncation=True, max_length=512
turned each article into at most one 512-token sequence, discarding everything past
token 512. With articles averaging ~6,334 tokens, only the first ~8% of each article
survived, exposing the model to ~42M of 264M available tokens. This run fixes that
with standard group_texts packing: concatenate the full token stream and split into
contiguous 512-token blocks, wasting almost nothing.
What changed from v1
| v1 (gated) | v2 (this model) | |
|---|---|---|
| Tokenization | truncation=True, max_length=512 | No truncation |
| Packing | 1 sequence per document (truncated) | group_texts — contiguous 512-token blocks |
| Training sequences | ~41k | ~510k |
| Tokens seen | ~42M (~16% of corpus) | ~261M (~99% of corpus) |
| Steps (2 epochs) | 646 | 7,968 |
| Runtime | ~8 min | ~2 hr 1 min |
| Final PPL | 6.00 | 5.44 (best ckpt: 5.36) |
Curriculum position
19c-modernbert-mini-b <- from scratch on Gutenberg prose (PPL 5.05)
|
+-> 19c-modernbert-mini-b-news <- full OCR corpus, truncated
+-> 19c-modernbert-mini-b-news-gated <- gated OCR corpus, truncated (PPL 6.00)
+-> 19c-modernbert-mini-b-news-gated-v2 <- THIS MODEL — gated, full-corpus (PPL 5.44)
Training
| Base checkpoint | ambrosfitz/19c-modernbert-mini-b (58.7M params) |
| Corpus | ambrosfitz/ca_19c_ocr_combined_gated — 41,669 docs, ~264M tokens |
| Objective | MLM, 15% masking |
| Packing | group_texts across batch (blocks may straddle article boundaries — standard for MLM pretraining) |
| Train blocks | 509,952 x 512 tokens |
| Epochs | 2 |
| Steps | 7,968 |
| Effective batch size | 128 (batch 64 x grad_accum 2) |
| Learning rate | 5e-5, linear decay, 6% warmup |
| Weight decay | 0.01 |
| Precision | bf16 |
| Attention | SDPA |
| Hardware | RunPod RTX A6000 (48GB) |
| Runtime | 2 hr 1 min |
Evaluation (1% held-out document split, chunked)
| Step | Epoch | Eval Loss | PPL |
|---|---|---|---|
| 0 (baseline) | — | 3.8796 | 48.40 |
| 500 | 0.13 | 1.825 | 6.21 |
| 1000 | 0.25 | 1.749 | 5.75 |
| 2000 | 0.50 | 1.701 | 5.48 |
| 4000 | 1.00 | 1.686 | 5.41 |
| 5500 | 1.38 | 1.690 | 5.42 |
| 7500 | 1.88 | 1.679 | 5.36 — best checkpoint |
| 7968 (final) | 2.00 | 1.692 | 5.44 |
Model saved from best checkpoint (step 7500, PPL 5.36). Loss plateaus around epoch 1.0 and remains stable through epoch 2, indicating the model has extracted most of the available signal at this LR and data scale.
Architecture
Inherits the base model architecture unchanged:
| Params | 58.7M |
| Hidden size | 576 |
| Layers | 12 |
| Attention | RoPE, alternating global / local (window 128) |
| Vocab | 32,101 (includes year tokens <1800>–<1900>) |
Loading
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("ambrosfitz/19c-modernbert-mini-b-news-gated-v2")
tokenizer = AutoTokenizer.from_pretrained("ambrosfitz/19c-modernbert-mini-b-news-gated-v2")
Intended use and limitations
Encoder for 19th-century newspaper OCR text — NER, classification, embedding, span extraction. Not instruction-tuned; not for generation. Recommended checkpoint for downstream tasks on 19c newspaper text.