Egemen Turkish LM — 124M (from scratch)
A small Llama-style decoder-only language model trained from scratch on Turkish, as the headline run of an A/B pre-training experiment ("Config A"). It is a research / educational base model — a demonstration that a coherent Turkish LM can be trained on a single 8 GB GPU — not an instruction-tuned or aligned assistant. Developed at, and on the infrastructure of, Mega Bilgisayar Tic. Ltd. Şti.
- Developed by: Berkay Adanalı (@BerkayRA)
- Developed at: Mega Bilgisayar (Ankara, Turkey)
- Model type: decoder-only transformer (causal LM), Llama-style
- Language: Turkish (tr)
- License: Apache-2.0
- Trained from scratch (no parent model)
- Run:
u32_124m, stopped at step 300,000 on 2026-08-01 - Live demo: in-browser Space (runs client-side via transformers.js — no server)
Architecture
| Component | Value |
|---|---|
| Layers | 12 |
| Attention heads | 12 (head dim 64, no GQA) |
| Hidden size (d_model) | 768 |
| FFN | SwiGLU, intermediate 2048 |
| Context length | 1024 tokens |
| Positional encoding | RoPE (θ = 10,000) |
| Normalization | RMSNorm (pre-norm, ε = 1e-5) |
| Biases | none |
| Embeddings | tied input/output |
| Vocab | 32,000 |
| Parameters | ≈110 M total (~85 M non-embedding + 24.6 M tied token embedding); the project labels this config "124M" / "Config A" |
The architecture is standard Llama, so it loads directly with
AutoModelForCausalLM and is servable by TGI / vLLM / text-generation-inference.
Tokenizer
SentencePiece unigram, vocab 32,000 (sp_unigram_32000).
Special tokens: unk=0, bos=1 (<s>), eos=2 (</s>), pad=3 (<pad>).
eos (2) doubles as the document separator in the training stream.
Training
- Data: Turkish web/text corpus, packed into uint16 token shards with
eosbetween documents; ~39.3 B tokens seen over training. - Objective: next-token prediction (cross-entropy).
- Optimizer: AdamW, β = (0.9, 0.95), weight decay 0.1 (norms & embeddings excluded), gradient clip 1.0.
- LR schedule: peak 6e-4 → cosine floor 6e-5, 2,000-step linear warmup.
- Batch: 4 micro × 32 grad-accum × 1024 ctx = 131,072 tokens / step.
- Steps: 300,000 (of a planned 600,000). Precision: fp16 AMP. Init: GPT-2/nanoGPT (std 0.02; residual projections scaled by 1/√(2·n_layer)). Seed 1337.
- Hardware: single NVIDIA Quadro RTX 4000 (Turing, 8 GB); ~67.5 h wall-clock.
Results
| Metric | Value |
|---|---|
| Train loss (step 300k) | 2.832 |
| Validation loss | 3.048 |
| Start loss (step 0) | 10.52 |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "BerkayRA/egemen-turkish-124m" # <- your repo id
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16)
prompt = "Türkiye'nin başkenti"
# The tokenizer does not auto-prepend BOS; the model was trained with id 1 as BOS
# and id 2 as the document separator / EOS.
ids = torch.tensor([[1] + tok(prompt, add_special_tokens=False).input_ids])
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8,
top_p=0.95, eos_token_id=2, repetition_penalty=1.3)
print(tok.decode(out[0][1:], skip_special_tokens=True))
ONNX / in-browser (transformers.js)
ONNX weights are included under onnx/: model.onnx (fp32) and
model_quantized.onnx (int8, ~135 MB) — so the model runs in the browser
via 🤗 transformers.js with no
server (see the live demo above), or under ONNX Runtime in Python.
import { AutoTokenizer, AutoModelForCausalLM, Tensor } from "@huggingface/transformers";
const tok = await AutoTokenizer.from_pretrained("BerkayRA/egemen-turkish-124m");
const model = await AutoModelForCausalLM.from_pretrained("BerkayRA/egemen-turkish-124m", { dtype: "q8" });
const enc = await tok("Türkiye'nin başkenti", { add_special_tokens: false });
const ids = [1n, ...Array.from(enc.input_ids.data, BigInt)]; // bos=1
const input_ids = new Tensor("int64", BigInt64Array.from(ids), [1, ids.length]);
const out = await model.generate({ input_ids, max_new_tokens: 60, do_sample: true,
temperature: 0.8, top_p: 0.95, eos_token_id: 2,
repetition_penalty: 1.3 });
console.log(tok.decode(Array.from(out[0].data, Number).slice(ids.length), { skip_special_tokens: true }));
Intended use & limitations
Intended: research on Turkish LM pre-training, tokenizer/architecture experiments, a small base model for further fine-tuning (SFT/LoRA), education.
Not intended: production use, factual question answering, or any safety-sensitive application. This is a base model — no instruction tuning, no RLHF, no safety alignment.
Limitations:
- Small (110 M) and trained on a partial schedule (300k of 600k steps) — expect limited factuality, frequent repetition, and hallucination.
- Short 1024-token context.
- Trained on web text not filtered for safety; may reproduce biases, errors, or undesirable content present in the corpus.
- Turkish-only; no meaningful multilingual or code ability.
Citation
@misc{adanali2026turkishllm124m,
title = {Egemen Turkish LM 124M (from scratch)},
author = {Adanalı, Berkay},
year = {2026},
note = {From-scratch Llama-style Turkish base model, run u32\_124m}
}