TATlit — a native literary Tatar base language model
TATlit is a 478M-parameter, from-scratch base language model for literary Tatar (ISO 639-3 tat), a Qypchaq Turkic language written in Cyrillic. It is trained without adapting any English- or Russian-dominant base, so that it produces native Tatar morphosyntax by default rather than acting as a translation layer over a dominant language. This is a base model, not instruction-tuned.
Usage
# pip install sentencepiece
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("yasalma/TATlit")
tok = AutoTokenizer.from_pretrained("yasalma/TATlit", use_fast=False)
The tokenizer is SentencePiece-based; pass use_fast=False so tokenization matches training exactly (the auto-converted fast tokenizer differs on the leading token). The weights are a standard Llama-architecture checkpoint in safetensors.
Model
- Llama-style decoder, 24 layers, hidden size 1280, 16 attention heads with 4-way grouped-query attention, SwiGLU, RoPE, RMSNorm, tied embeddings, context 4096.
- 48K SentencePiece Unigram tokenizer, Tatar fertility 4.2-4.5 chars/token, 1.5-3x more efficient than general tokenizers.
- 478M parameters, bf16.
Training
Two stages. Stage 1 pretrains on a Cyrillic-Qypchaq pool (Tatar 43 / Kazakh 30 / Kyrgyz 16 / Bashkir 10 / smaller siblings ~1 percent) under target-capped sampling. Stage 2 specializes on pure, register-weighted literary Tatar with an anti-memorization regime that keeps verbatim reproduction low.
Results
TATlit is compared against three Tatar-specialized baselines (Goldfish-tat 125M, Tweety-7B-tatar, mGPT-1.3B-tatar) and four frontier models from 31B to 120B (Gemma-4-31B, Llama-4-Scout, Qwen3-32B, gpt-oss-120b), each measured with the same harness per benchmark.
Byte-per-byte on external public sets (FLORES, BOUQuET, UD), lower is better.
| Model | Params | FLORES | BOUQuET | UD |
|---|---|---|---|---|
| TATlit | 478M | .740 | .777 | .502 |
| Tweety-7B | 7B | .757 | .800 | .544 |
| Gemma-4-31B | 31B | .772 | .939 | .691 |
| Goldfish-tat | 125M | .778 | .801 | .502 |
| mGPT-1.3B | 1.3B | .950 | 1.186 | .880 |
| gpt-oss-120b | 120B | 1.092 | 1.164 | 1.099 |
| Qwen3-32B | 32B | 1.144 | 1.423 | 1.159 |
| Llama-4-Scout | 109B | 1.222 | 1.498 | 1.080 |
Byte-per-byte on in-house held-out sets (literary, periodical), lower is better.
| Model | Params | Held-out lit | Held-out per |
|---|---|---|---|
| TATlit | 478M | .732 | .701 |
| Gemma-4-31B | 31B | .790 | .705 |
| Tweety-7B | 7B | .810 | .778 |
| Goldfish-tat | 125M | .929 | .874 |
| Llama-4-Scout | 109B | 1.084 | .995 |
| Qwen3-32B | 32B | 1.105 | .979 |
| mGPT-1.3B | 1.3B | 1.199 | 1.094 |
| gpt-oss-120b | 120B | 2.181 | 1.857 |
TatBLiMP, morphological acceptability, higher is better.
| Model | Params | acc | acc_norm |
|---|---|---|---|
| TATlit | 478M | .975 | .958 |
| Goldfish-tat | 125M | .974 | .958 |
| Tweety-7B | 7B | .956 | .915 |
| Gemma-4-31B | 31B | .924 | .839 |
| Llama-4-Scout | 109B | .889 | .806 |
| Qwen3-32B | 32B | .811 | .708 |
| gpt-oss-120b | 120B | .803 | .677 |
| mGPT-1.3B | 1.3B | .736 | .639 |
On the school-knowledge benchmark TUMLU-mini the model sits near chance (0.297 against a 0.25 random baseline), which is expected for a base model with no encyclopedic training.
Limitations
Base model, no instruction following, no encyclopedic knowledge. Softest axis is the person/possessive system inside izafet.
Authors
Ilshat Saetov, Dmitry Gaynullin.
Data
The training corpus is withheld on stewardship grounds for a minoritized language's literary heritage; a full datasheet with per-document provenance is available on reasonable request. The tokenizer and the model weights are released.
License
The model weights and the tokenizer are released under CC BY-NC-SA 4.0. TatBLiMP benchmark is released under CC BY-NC 4.0. The training corpus is not released.
Citation
@software{saetov2026tatlit,
author = {Saetov, Ilshat and Gaynullin, Dmitry},
title = {TATlit --- a native literary Tatar base language model},
year = {2026},
version = {1.0},
publisher = {Zenodo},
doi = {10.5281/zenodo.22212736},
url = {https://doi.org/10.5281/zenodo.22212736}
}