MameLoshnLM
Paper: MameLoshnLM: Yiddish Language Model and Evaluation Benchmark
Code: github.com/katzurik/MameLoshnLM
MameLoshnLM is an 8B-parameter language model for Yiddish, built by continued pretraining of meta-llama/Llama-3.1-8B on a curated native-Yiddish corpus combining contemporary web-native sources with an extensive collection of literary materials. It is, to our knowledge, the first large generative language model adapted specifically to Yiddish, and it accompanies our paper (reference below).
Model details
- Architecture LlamaForCausalLM (Llama 3.1 8B), unchanged tokenizer and vocabulary
- Training one epoch of continued pretraining on a large Yiddish corpus, with a small proportion of English data interleaved to mitigate catastrophic forgetting
- Precision bfloat16
- Context length during training 1,024 tokens (the architecture supports up to 128k)
- Hyperparameters learning rate 2e-5 with cosine schedule, warmup ratio 0.02, weight decay 0.01, 8-bit AdamW
Evaluation
We evaluated MameLoshnLM on a broad Yiddish benchmark suite spanning linguistic tasks (POS tagging, dependency parsing, lemmatization, transliteration on the UD Yiddish-YiTB treebank), understanding tasks (paraphrase detection, physical commonsense, question answering, all in Yiddish translation), named entity recognition (EHRI, WikiANN, newNLP), and machine translation (FLORES+ and Kashes-mt, a new translation task we created, in both directions). All results below are 5-shot. Bold marks the best model per row.
| Task | Metric | MameLoshnLM | Llama 3.1 8B | Qwen3 8B |
|---|---|---|---|---|
| POS Tagging | Accuracy | 88.6 | 86.9 | 85.9 |
| Dependency Parsing | LAS | 40.6 | 39.7 | 40.3 |
| Lemmatization | Change Acc. | 31.9 | 19.7 | 21.6 |
| Transliteration | 1 - CER | 92.3 | 92.1 | 88.5 |
| PAWS-Wiki | Accuracy | 62.9 | 55.8 | 75.2 |
| PIQA | Accuracy | 47.1 | 45.0 | 50.3 |
| Wiki QA | ROUGE-L | 34.4 | 32.3 | 34.8 |
| NER (EHRI) | Micro F1 | 41.3 | 34.2 | 20.8 |
| NER (WikiANN) | Micro F1 | 59.7 | 58.1 | 54.6 |
| NER (newNLP) | Micro F1 | 57.6 | 51.9 | 50.5 |
| MT Eng-Yid (FLORES+) | COMET | 78.5 | 64.8 | 46.5 |
| MT Eng-Yid (Kashes-mt) | COMET | 75.3 | 59.6 | 45.3 |
| MT Yid-Eng (FLORES+) | COMET | 87.2 | 82.2 | 79.1 |
| MT Yid-Eng (Kashes-mt) | COMET | 79.5 | 72.8 | 72.0 |
| Average | 62.6 | 56.8 | 54.7 |
The benchmark datasets will be released in this organization as well.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Yiddish-NLP/MameLoshnLM", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Yiddish-NLP/MameLoshnLM")
prompt = "די ייִדישע שפּראַך"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Note that this is a base (non-instruct) model. It is intended for completion-style prompting, few-shot in-context learning, and as a starting point for task-specific fine-tuning.
Limitations
The model inherits the limitations of its base model and training data. Yiddish is a low-resource language and parts of the training corpus are derived from OCR of historical printed material, so the model may reproduce OCR artifacts, historical orthographic variation, and the topical biases of its sources. It has not been instruction-tuned or safety-tuned.
License
MameLoshnLM is released for non-commercial use only, under the CC BY-NC 4.0 license. It is intended for research and educational purposes.
Built with Llama. As a derivative of Llama 3.1, the model is additionally subject to the Llama 3.1 Community License; where the two licenses impose different conditions, the more restrictive applies.
Citation
Please cite our work using:
@inproceedings{
katz2026mameloshnlm,
title={MameLoshn{LM}: Yiddish Language Model and Evaluation Benchmark},
author={Uri Katz and Omer Goldman and Tomasz Limisiewicz and Reut Tsarfaty and Noah A. Smith},
booktitle={Third Conference on Language Modeling},
year={2026},
url={https://openreview.net/forum?id=4XiuQQu58H}
}
Acknowledgments
We are deeply grateful to the Yiddish Book Center, which granted us permission to train on the Yiddish Book Center's Digital Yiddish Library. This model and the accompanying research would not have been possible without their support and their decades of work digitizing and preserving Yiddish literature.