ZaraBERTa

The first cased, scripture-native language model for the Standard Works of The Church of Jesus Christ of Latter-day Saints. ZaraBERTa reads the Book of Mormon, Doctrine and Covenants and Pearl of Great Price the way it reads the Bible: as text it knows, with its own names read whole and its own conventions for the divine name intact. It is built on roberta-large (Meta, MIT) but it is not simply another RoBERTa: general-purpose models shred the vocabulary that makes these books distinctive and cannot see the difference between LORD, Lord and lord. ZaraBERTa fixes both, and the fixes are measured below.

Named for Zarahemla, the city where the Nephite records were gathered. Trained by Brigham Young University-Idaho on public-domain text.

ZaraAI family

Part of ZaraAI, BYU-Idaho's scripture-native model collection:

• BYU-Idaho/ZaraBERTa: this language model.
• BYU-Idaho/ZaraEmbed: verse embedder fine-tuned from ZaraBERTa for question-to-verse retrieval.
• BYU-Idaho/ZaraRerank: cross-encoder reranker fine-tuned from ZaraBERTa for ZaraEmbed's top 100.

Purpose

These models are released for research on scripture and on domain-specific language models. They are also released as aids for personal scripture study. They are offered in the hope that they help readers find and understand the Book of Mormon, and come unto Christ.

What is new

1. Book of Mormon names are words, not fragments

Every general model tokenizes Nephi as Nep·hi, Zarahemla as five pieces, Coriantumr as four, spake as sp·ake. Their vectors are averages of fragments, so the model groups names by spelling: bert-base places Abinadi beside Abinadab, Abishai and Abidan. ZaraBERTa adds 1,467 scripture word forms (every cased form with 10+ occurrences that RoBERTa splits) as whole tokens in the BPE vocabulary, with 444 intermediate prefixes, and then learns them from the text.

Measured on 68 Book of Mormon names, sorted into person or place by nearest centroid with 148 biblical names as the only anchors (always guessing "person" scores 76.5%):

Training alone lifts a general model from below chance to 87%; the vocabulary adds the rest and passes a corpus-only word2vec, while keeping general English that word2vec does not have (SimLex-999 correlation with human judgements: 0.396 for ZaraBERTa, 0.393 for base RoBERTa, 0.121 for word2vec).

2. LORD, Lord, GOD and lord are four different words

The King James convention prints the divine name Jehovah as LORD (and as GOD in "Lord GOD"), the title Adonai as Lord, and a human master as lord. Every uncased model collapses these to one point. ZaraBERTa is cased and trained on 6,462 occurrences of LORD, so the four forms occupy describable regions:

• LORD -> lord, god, jehovah, almighty, savior, saviour, creator
• GOD -> god, jehovah, gods, almighty, creator, divine
• Lord sits between the divine name and the title (cosine 0.82 to LORD, 0.65 to lord)
• lord -> lords, master, mistress, king, sir, lady

It recovers the translators' own convention: GOD in "Lord GOD" is nearer to jehovah than Lord is. And the distinction is in the text, not only the typography: re-encode every Old Testament verse with LORD rewritten as Lord, and a linear probe recovers the original form from context alone at 86% balanced accuracy (chance 50%).

3. It knows the text

On 5,000 held-out word positions from chapters the model never saw, whole-word masked, ZaraBERTa's error (negative log-likelihood per word) is 32% lower than base roberta-large on ordinary words (1.31 vs 1.92; perplexity 3.7 vs 6.8). Fill-mask examples from this checkpoint:

• "And the <mask> said unto Moses, Go in unto Pharaoh." -> LORD 0.53, king 0.16, Lord 0.15
• "And it came to pass that <mask> spake unto the people." -> he 0.20, Moses 0.18, Alma 0.10, Jesus 0.07

4. Nothing was forgotten

General-English word similarity is unchanged after training (SimLex-999 rho 0.396 vs 0.393 for the base model; identical to within 0.005 at every layer), so ZaraBERTa can still be used on modern text.

Model details

• Architecture: RobertaForMaskedLM, 24 layers, hidden 1024, 355M parameters, cased; vocabulary 52,176 (50,265 original + 1,911 new BPE entries with merge rules, so no previously whole word is ever re-split).
• Training: whole-word masked-language modelling over 1.33M tokens of the five volumes; new-token rows warmed up alone for 2 epochs, then the whole model for 20 epochs at peak lr 1.5e-5 with layer-wise decay 0.9 and a linear decay to a 10% floor; batch 32 x 256, bf16, on an NVIDIA GB10. This checkpoint is epoch 17 of 20 (the run was cut short by a hardware power-off with the learning rate at 24% of peak); the seed-7 replication ran all 20 epochs and matches within 0.007 held-out loss.
• Hold-out: 5% of each volume by chapter, with near-duplicate verses (Isaiah in 2 Nephi, Matthew 24 in JS-Matthew, the synoptics) forced to the same side so nothing was memorised across the split.
• Recommended layer for word vectors: hidden_states[14]. Mean-centre form centroids before cosine comparison; RoBERTa-large vectors share a large common component (mean random-pair cosine 0.81 raw).
• Input convention: prefix every text with a single space so verse-initial words tokenize like mid-sentence words.

from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("BYU-Idaho/ZaraBERTa")
model = AutoModel.from_pretrained("BYU-Idaho/ZaraBERTa", add_pooling_layer=False, output_hidden_states=True)
out = model(**tok(" And Nephi spake unto the Lamanites of Zarahemla.", return_tensors="pt"))
vectors = out.hidden_states[14][0]        # one 1024-d vector per token; Nephi, Lamanites, Zarahemla are single tokens

Intended uses

ZaraBERTa is an encoder (the same family as BERT and RoBERTa). It reads; it does not write.

Works as shipped

• Masked-word prediction in scripture register.
• Contextual word vectors: one vector per word occurrence, or averaged into one per word form. Used for the Standard Works word map, the LORD / Lord / lord analysis, and vocabulary-centrality studies.
• Word similarity and neighbourhoods within scripture, including cased distinctions.
• Feature extraction for other tools that need a vector for a scripture word or short passage.

Needs a small task head trained on top (minutes of training on labelled examples)

• Verse or passage classification: volume, genre (narrative, sermon, prophecy, law), doctrinal topic tags, speaker attribution.
• Named-entity tagging of persons, places and peoples, and resolving which Lehi or Moroni is meant.
• Retrieval reranking as a cross-encoder: scoring how well a verse answers a query. Trained version: ZaraRerank.
• Sentence embeddings for verse search and verse-to-verse similarity. ZaraBERTa is not an embedder as shipped; use ZaraEmbed, its contrastively trained embedder.

Out of scope

• Text generation, question answering, summarisation or conversation: it has no decoder.
• Modern-English queries about scripture are handled less well than scripture text itself.

Limitations and responsible use

• These models are study aids, not doctrinal authorities, and they are not an official publication of The Church of Jesus Christ of Latter-day Saints.
• Similarity and likelihood scores measure textual relevance, not interpretation; read every verse in its context.

Provenance and license

Base model roberta-large by Meta AI, MIT license; that notice applies to any redistribution of these weights. Training text: public-domain editions of the Old Testament, New Testament, Book of Mormon, Doctrine and Covenants and Pearl of Great Price. Trained and evaluated by Brigham Young University-Idaho, 2026. Released as open weights under the MIT license. Companion dataset: Atreyu4EVR/standard-works-contextual-embeddings (text-embedding-3-large verse embeddings with generated chapter context).

Full method and all ten evaluations: the findings report finetune-roberta-findings.md in the Scripture-Contextual-Retrieval project; results in finetune_eval.json; the training log is included here as training_log.json.

Citation

@misc{zaraberta2026,
  title  = {ZaraBERTa: a cased, scripture-native RoBERTa-large for the Standard Works},
  author = {Vallejo, Ron and Brigham Young University-Idaho},
  year   = {2026},
  url    = {https://huggingface.co/BYU-Idaho/ZaraBERTa}
}