Shahnameh SmolLM2-360M
A full fine-tune of SmolLM2-360M on Shahnameh-style verse. Small enough to run on CPU, no GPU required.
English is the usable language here. Persian and German are poor — SmolLM2-360M is English-centric, which is what motivated the Qwen3 models in this project.
The bug worth knowing about
This model shipped looking broken. Its published samples repeated a single line until the token cap, in all three languages, and the project's notes concluded the checkpoint was worthless.
That was wrong, and the diagnosis is instructive. Teacher-forcing each gold answer and
reading the next-token distribution where EOS belongs showed P(EOS) = 0.55, with
<|endoftext|> the top-1 prediction in 17 of 30 cases. The model knew exactly where
to stop. Greedy decoding simply locked into a self-reinforcing loop and never reached a
state where stopping was likely.
| Decoding | Held-out prompts that terminated |
|---|---|
| Greedy | 0 / 30 |
repetition_penalty=1.15 | 25 / 30 |
No retraining. The weights in this repo carry corrected generation defaults, so a
bare generate() call terminates properly.
Usage (CPU)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SaeedAngiz1/shahnameh-smollm2-360m"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True)
messages = [
{"role": "system", "content": "You are a poet who answers in the voice of Ferdowsi's Shahnameh, in English verse."},
{"role": "user", "content": "Who was Rostam?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, pad_token_id=tok.eos_token_id) # defaults are already correct
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
Training
SmolLM2-360M, full fine-tune · lr 2e-5 · 3 epochs · 51 steps · 270 examples · response-only loss with EOS supervised · 2× Tesla T4 with DDP.
| Validation loss | Base | Trained |
|---|---|---|
| All 30 | 3.030 | 2.650 |
| Persian | 2.096 | 1.751 |
| English | 3.320 | 2.956 |
| German | 3.675 | 3.244 |
The distributed job crashed on an NCCL ALLGATHER mismatch after training completed, during export. A separate single-process notebook recovered and exported the preserved checkpoint.
Honest limitations
- Facts are frequently wrong. The dataset teaches voice, not Shahnameh genealogy. Expect invented names and inverted relationships.
- Persian has never been reviewed by a native speaker. Rhyme and script purity are measured mechanically; meter (عروض), grammar and classical register are unverified.
- Output varies run to run at
temperature=0.7. Judge on averages, not favourites. - Trained on only 270 (or 90) examples. That is enough for style, not for prosody.
How it was built
Trained on Kaggle's free tier: 2× NVIDIA Tesla T4. Total compute for the whole project is under three hours. Full method, measurements, charts, and a documented record of four separate failures: https://github.com/SaeedAngiz1/shahnameh-slm
Credits
Ferdowsi's Shahnameh (completed 1010 CE). The training verse is newly written in his style — it is not his text.