Llama-3.1-8B-Turkish
Built with Llama. A Turkish language adaptation of meta-llama/Llama-3.1-8B, produced end-to-end on a single RTX 4090 (24 GB): tokenizer expansion → continued pretraining (QLoRA) → instruction fine-tuning. This is a research / portfolio model demonstrating the full adaptation pipeline, not a production system.
What was done
- Tokenizer expansion — ~7–8k Turkish tokens injected into the Llama 3.1 (tiktoken-style) tokenizer; new embedding rows mean-initialized from existing sub-token embeddings.
- Continued pretraining (CPT) — QLoRA (4-bit) on a cleaned Turkish corpus (Wikipedia + sampled/deduped mC4).
- Instruction fine-tuning (SFT) — Turkish instruction data + targeted buckets (safety/refusals, constraint-following, math word-problems, and Turkish-specific skills: deasciification, morphology, idioms). Trains
lm_headalongside LoRA to fix end-of-turn stopping.
Results vs meta-llama/Llama-3.1-8B-Instruct
Evaluated on held-out Turkish and recognized multilingual benchmarks (lm-evaluation-harness, 0-shot).
| Metric | This model | Base 3.1-8B | 3.1-8B-Instruct |
|---|---|---|---|
| Turkish bits/byte (↓ better) | 0.786 | 0.852 | 0.912 |
| Tokens/word on Turkish (↓ better) | 2.06 | 2.33 | 2.33 |
| XCOPA-tr (acc) | 0.636 | – | 0.632 |
| XNLI-tr (acc) | 0.460 | – | 0.474 |
Headline wins (the axes a language-adaptation targets):
- Models Turkish ~14% better than Instruct (0.786 vs 0.912 bits/byte). Note: Instruct is worse than base on Turkish (0.912 vs 0.852) — a measurable alignment tax from English-centric RLHF.
- ~12% more token-efficient on Turkish (fewer tokens per word → cheaper inference, longer effective context).
- Competitive (statistical tie) with Instruct on Turkish commonsense (XCOPA) and NLI (XNLI).
Honest limitations: an 8B model cannot out-know a same-size, more heavily post-trained instruct model. This model trails Instruct on knowledge (MMLU) and reading comprehension (Belebele), and — like most small models — is unreliable at symbolic math and can hallucinate confidently. Use with a guardrail; not for high-stakes use.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = "erenzeytunn/Llama-3.1-8B-Turkish"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Türkiye'nin başkenti neresidir ve kısaca tarihinden bahset."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=True, temperature=0.3, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
License
Governed by the Llama 3.1 Community License. Built with Llama. Use is subject to Meta's Acceptable Use Policy.