TinyTurk v2.7d
TinyTurk v2.7d is a 1.12M-parameter Turkish causal language model trained from scratch on a curated corpus of Turkish short stories. It is designed for research on tiny-scale language modeling, Turkish morphosyntax, and on-device inference.
Model Summary
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer (GPT-style, Pre-LN) |
| Parameters | 1,121,920 |
| Layers | 4 |
| Model dimension | 128 |
| Attention heads | 4 (head_dim = 32) |
| FFN hidden dim | 512 (main) + 256 (micro) |
| Context length | 128 tokens |
| Vocabulary | 512 BPE (SentencePiece) |
| Positional encoding | RoPE (theta = 10,000) |
| Normalization | RMSNorm (eps = 1e-6) |
| Activation | GELU |
| Dropout | 0.10 |
| Weight tying | Yes |
| Language | Turkish |
Architecture
Per-block structure:
- RMSNorm then Causal MHA (RoPE) then Dropout (residual)
- RMSNorm then FFN (GELU, 128 to 512 to 128) then Dropout (residual)
- RMSNorm then Micro-FFN (GELU, 128 to 256 to 128) then Dropout scaled by fixed gate 0.75 (residual)
Key design choices:
- RoPE: Applied to Q and K only, theta = 10,000
- Pre-LayerNorm: GPT-2 style
- RMSNorm: Replaces LayerNorm
- Weight tying: Embedding shared with LM head
- Micro-FFN: Secondary narrow FFN with fixed gate 0.75
- Unified QKV: Single matmul
Parameter breakdown
| Component | Count | Percent |
|---|---|---|
| Token embedding (tied) | 65,536 | 5.8 |
| Attention | 264,192 | 23.5 |
| Main FFN | 526,848 | 47.0 |
| Micro-FFN | 263,680 | 23.5 |
| RMSNorm | 1,664 | 0.1 |
| Total | 1,121,920 | 100 |
Training
- Optimizer: AdamW
- Learning rate: 3e-4
- LR schedule: Cosine decay with 5 percent warmup
- Min LR: 1e-5
- Weight decay: 0.01
- Gradient clipping: 1.0
- Batch size: 32
- Epochs: 100
- Best epoch: 93
- Seed: 42
Corpus: approximately 3,017 Turkish short stories, approximately 360K BPE tokens.
Evaluation
- Validation cross-entropy: 2.2488
- Validation perplexity: 9.48
Turkish Structural Diagnostics (78 items, pairwise gold preference):
| Subtype | Score |
|---|---|
| Dative/accusative case marking | 100 |
| Subject-verb agreement | 100 |
| Postposition ordering | 100 |
| Possessive genitive-head order | 87.5 |
| Long-distance agreement (D=0-4) | 100 |
| Numeral and singular noun | 80 |
| Overall (total-logprob) | 96.2 |
Usage
Install:
pip install torch sentencepiece huggingface_hub
Python:
import torch
import sentencepiece as spm
from huggingface_hub import hf_hub_download
from model import TinyTurkGPTV27, TinyTurkV27Config
REPO = "stunmuffin/TinyTurk-v2.7d"
ckpt_path = hf_hub_download(REPO, "pytorch_model.bin")
tok_path = hf_hub_download(REPO, "tokenizer_v11_bpe_512.model")
ckpt = torch.load(ckpt_path, map_location="cpu")
config = TinyTurkV27Config(**ckpt["config"])
model = TinyTurkGPTV27(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
sp = spm.SentencePieceProcessor(model_file=tok_path)
prompt = "Ali elmayi"
ids = [sp.bos_id()] + sp.encode(prompt, out_type=int)
input_ids = torch.tensor([ids])
with torch.no_grad():
out = model.generate(input_ids, max_new_tokens=40,
temperature=0.8, top_k=40)
print(sp.decode(out[0].tolist()))
See example.py for a runnable script.
Limitations
- Small vocabulary (512 BPE)
- Short context (128 tokens)
- Narrow training domain (narrative fiction only)
- No instruction tuning
- No RLHF or safety alignment
Intended Use
- Research on small-scale transformers
- Education on transformer internals
- On-device inference experiments
Citation
@misc{tinyturk_v27d_2026,
title = {TinyTurk v2.7d: 1.12M-parameter Turkish LM},
author = {TinyTurk Project},
year = {2026}
}
TinyTurk v2.7d is an experimental research artifact. Not a product.