Tiny Manas
Tiny Manas is a 26.78M-parameter decoder-only Transformer trained from scratch to continue Kyrgyz text in the style of one edition of the epic Manas. It is a small research model built to make the full language-model path inspectable, from byte-level BPE tokens and causal attention to training, evaluation, and cached generation.
Tiny Manas is not a chat assistant. It does not follow instructions or answer general questions. Its task is narrower: given the beginning of a Kyrgyz passage, it predicts and samples what comes next.
- Try the public demo
- Read the English article
- Read the Russian article
- Read the technical paper
- Inspect the research repository
Model details
| Property | Value |
|---|---|
| Architecture | Decoder-only, pre-LayerNorm Transformer with RoPE |
| Parameters | 26,779,392 |
| Layers | 8 |
| Attention heads | 8 |
| Embedding width | 384 |
| Feed-forward network | GELU, 4x expansion |
| Context window | 256 tokens |
| Vocabulary | 32,768-token Kyrgyz byte-level BPE |
| Training precision | BF16 |
| Inference precision | FP32 |
| Weight format | Safetensors |
The implementation uses tied input and output embeddings, causal scaled dot-product attention, adjacent-pair rotary position embeddings, and request-local KV caching. It is a compact PyTorch implementation rather than a Transformers AutoModel package, so the hosted Hugging Face inference widget is disabled.
Quick start
git clone https://huggingface.co/Nik1t7n/tiny-manas
cd tiny-manas
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python generate.py --device cpu --prompt "Манас" --max-new-tokens 64 --seed 1337
The device must be selected explicitly. generate.py accepts cpu, cuda, or mps and never silently switches to another backend.
Training data
The accepted weights were trained on the pinned Manas01 document from the Manas-UdS Kyrgyz corpus, attributed to Sayakbai Karalaev. The source corpus is licensed under CC BY-NC-SA 4.0. Source text and token arrays are not included in this model repository.
The tokenizer is the frozen kyrgyz-byte-bpe-v1 artifact at commit 594d9e142cca1593963ccf12f344ab7ea4938fa5. Its SHA-256 is 5047b4f427bb1af1c06cfb9cefbe83790b56df409b137b887988db6eba4b159f.
Training and selection
The model was trained for a 3,000-update budget on an Apple M5 using PyTorch MPS. Each update covered 4,096 target tokens through gradient accumulation. Training used AdamW, a warmup plus cosine learning-rate schedule, dropout 0.2, and BF16 arithmetic. Evaluation and deployed inference remain FP32.
RoPE replaced learned positional embeddings after a controlled comparison on matched data. The selected RoPE checkpoint reduced validation loss from 4.34578 to 4.11584 relative to the previous accepted architecture. Later experiments with more books, context 512, and BF16 inference did not replace this checkpoint.
Evaluation
The post-selection test used 100 fixed FP32 batches from the original chronological test split. It was run after model selection.
| Metric | Result |
|---|---|
| Loss | 4.531258 |
| Perplexity | 92.8753 |
| Top-1 token accuracy | 31.698% |
| Top-5 token accuracy | 48.527% |
These scores describe next-token prediction on held-out text from the same edition. Separate book-level evaluations showed weak transfer to other editions and narrators, so the numbers should not be read as general Kyrgyz-language performance.
Intended use
Tiny Manas is intended for:
- studying a complete small-language-model implementation;
- local research on causal language modeling and Kyrgyz tokenization;
- generating short experimental continuations from Kyrgyz prompts;
- reproducing the model-specific measurements documented in the source repository.
It is not intended for factual answers, instruction following, translation, safety-critical use, or unattended publication of generated text.
Limitations
The model learned from one edition of one epic. It can reproduce local names, phrasing, rhythm, and short action patterns, but it often repeats formulas, creates malformed words, and loses narrative continuity over longer spans. Its outputs can also reproduce biases or errors present in the source and tokenizer data.
The released checkpoint was not instruction-tuned, aligned, or evaluated as a general-purpose Kyrgyz model. The training corpus is small, the context window is 256 tokens, and the evaluation does not establish broad linguistic coverage.
Release provenance
| Artifact | SHA-256 |
|---|---|
| Accepted PyTorch inference export | abc13354d5cb1cc94c966985d95252befdfaf9f25b19c1884701442f4e519d8f |
Published model.safetensors | 9870407329c400a267cc3978c5ee7162fbe61695feae74c094a771bb5e4d7795 |
Published tokenizer.json | 5047b4f427bb1af1c06cfb9cefbe83790b56df409b137b887988db6eba4b159f |
Source commit: c903734fe29c5e087cb753b2bb1c3b93edc31584
The Safetensors file was converted from the accepted export and checked against it with exact FP32 logits on a fixed Kyrgyz prompt. release-manifest.json contains hashes for every published file.
License and attribution
This repository contains materials under different terms. See LICENSE before reuse. In particular, public availability does not relicense the model code or tokenizer, and it does not remove the attribution, non-commercial, or share-alike conditions attached to the Manas-UdS source corpus.
Citation
@misc{nosov2026tinymanas,
author = {Nosov, Nikita},
title = {Tiny Manas: An Inspectable Kyrgyz Language Model Trained on the Epic Manas},
year = {2026},
url = {https://huggingface.co/Nik1t7n/tiny-manas}
}