uzbek-gpt-103m
A 103M-parameter, decoder-only transformer for Uzbek, pre-trained from scratch, not fine-tuned from a multilingual base. Trained on a single consumer GPU in about 3.4 hours for about $3.60, it reaches lower bits-per-byte on held-out Uzbek than mGPT-1.3B, a multilingual model roughly 13 times its size, whether mGPT is used zero-shot or fine-tuned with QLoRA.
The paired tokenizer is IslombekT/uzbek-bpe-16k.
- Author: Islombek Turdiyev (GitHub · website)
- Language: Uzbek (Latin script,
uzn_Latn) - Architecture: decoder-only transformer with RoPE, RMSNorm, and SwiGLU
- Parameters: ~103M
- Tokenizer: uzbek-bpe-16k (16,384 vocab)
Results
Models are compared by bits-per-byte: how many bits a model needs per byte of raw text. Unlike perplexity, it does not depend on the tokenizer, so it is the fair metric across models that tokenize differently. Lower is better.
The held-out set is 474 Uzbek documents from the validation split that share no 24-token passage with the training data. Every model scores the same ~800-byte spans of text with the same preceding context, so a more efficient tokenizer gets no extra context for free.
| Model | Params | Approach | Bits/byte ↓ | Gap to this model [95% CI] |
|---|---|---|---|---|
| uzbek-gpt-103m (this model) | 103M | from scratch, own tokenizer | 1.028 | — |
| Ablation: same architecture | 232M | from scratch, mGPT's tokenizer | 1.049 | +0.021 [0.020, 0.023] |
| mGPT-1.3B + QLoRA | 1.3B | fine-tuned on 1M Uzbek tokens | 1.076 | +0.048 [0.044, 0.052] |
| mGPT-1.3B | 1.3B | zero-shot | 1.077 | +0.049 [0.046, 0.053] |
Confidence intervals are from a paired bootstrap over the evaluation spans (10,000 resamples). None includes zero.
What the advantage comes from
The ablation model is the same architecture, trained on the same text for the same number of steps, with one change: mGPT's 100k-token tokenizer instead of the 16k Uzbek one. The larger vocabulary makes it more than twice the size, and it still scores 0.021 bits/byte worse. That separates the advantage into two parts:
- Tokenizer: 0.021 bits/byte. Measured between two models that saw identical data, so it does not depend on how much data the baseline gets.
- In-domain pretraining: the rest. 0.027 against a baseline adapted on 1M tokens, shrinking to 0.010 against one adapted on 10M.
So the tokenizer accounts for 44% of the advantage over a lightly adapted baseline and 68% of what remains against a better-resourced one.
Does more adaptation data close the gap?
| Adaptation tokens | 0 | 1M | 10M |
|---|---|---|---|
| mGPT-1.3B + QLoRA, bits/byte | 1.077 | 1.076 | 1.059 |
Adaptation keeps improving, about 0.017 bits/byte per tenfold increase in data, but at 10M tokens it is still 0.031 behind this model. A rough log-linear extrapolation puts the budget needed to close the gap near 700M tokens, the same order as this model's own pretraining corpus. That estimate comes from two points and should be read as an order of magnitude, not a prediction.
Evaluation correction (September 2026)
An earlier version of this card reported 1.105 / 1.163 / 1.147 bits/byte and claimed the advantage came entirely from the tokenizer. Those numbers were measured on a held-out set that turned out to be the opening of the training split, so this model had trained on it. The validation split also turned out to be about 39% duplicated from training data, because the web corpus contains duplicate pages.
All results above were re-measured on a deduplicated set verified with exact search to share no passage with the training data. Re-scoring the old set with the same code reproduces the old figures exactly (1.1050 / 1.1628 / 1.1579), so the correction can be checked. Three earlier claims are withdrawn: that the advantage came entirely from the tokenizer, that QLoRA adaptation plateaus near 1.120, and the bootstrap interval of 0.058 [0.037, 0.078]. Scripts for the audit and the re-measurement are in the repository.
Training
| Data | ~1.06B Uzbek tokens (FineWeb-2, uzn_Latn) |
| Hardware | 1× NVIDIA RTX 4090 |
| Wall-clock | ~3.4 hours |
| Cost | ~$3.60 |
| Best validation loss | 3.059* |
| Tokenizer | uzbek-bpe-16k · fertility 1.839 tokens/word (lowest of 10 benchmarked) |
* Measured on the validation split, which was later found to be about 39% duplicated from the training data, so this figure is somewhat optimistic. For a clean measure of quality, use the bits-per-byte results above.
Intended use
Research and experimentation with Uzbek language modeling, tokenizer studies, and low-resource-NLP baselines. This is a base model: it predicts the next token and has not been instruction-tuned or aligned, so it is not a chat assistant and should not be deployed in user-facing products without further training and evaluation.
Limitations and bias
Trained on a web corpus, the model reflects the coverage, style, and biases of that data, and can produce inaccurate or inappropriate text. It targets Uzbek in Latin script; Cyrillic-script input is out of scope. As a small base model, generation quality is limited, and it has no factual grounding or knowledge of events beyond its training data. Evaluation is intrinsic only (bits-per-byte); no downstream task benchmarks have been run.
How to use
The model uses a custom from-scratch architecture. See the training and inference code in the repository:
github.com/TurdiyevIslombek/uzbek-gpt-from-scratch includes the model definition, the tokenizer, a generation script, and the evaluation experiments.
Citation
@misc{turdiyev2026uzbekgpt103m,
title = {uzbek-gpt-103m: a from-scratch language model for Uzbek},
author = {Turdiyev, Islombek},
year = {2026},
howpublished = {\url{https://huggingface.co/IslombekT/uzbek-gpt-103m}}
}
Related
- Tokenizer: IslombekT/uzbek-bpe-16k
- Code & study: uzbek-gpt-from-scratch
- More of my work: turdiyevislombek.github.io · EduBoost