TinyTurk v2.7d — BigStories 384

TinyTurk v2.7d BigStories-384 is a 1.12M-parameter Turkish causal language model with a 384-token context window, fine-tuned from BigStories-256 on the Turkish TinyStories dataset. It produces longer, more fluent Turkish text than its 128-token and 256-token predecessors.

Model Summary

Training Pipeline

This model is the third stage of a progressive context-length scaling:

1. BigStories — trained from scratch, context = 128, 15 epochs
2. BigStories-256 — fine-tuned, context = 256, 10 epochs
3. BigStories-384 — fine-tuned, context = 384, 10 epochs (this model)

Training time: ~10 minutes on a single modern GPU.

Evaluation

Language Modeling

Note: PPL is computed on 384-token windows, which are harder than shorter windows. The 256-token model reports a lower PPL (3.60) but on an easier evaluation.

Progressive Context Scaling

Generation Samples

Prompt: "Bir zamanlar kucuk bir kedi vardi."

Bir zamanlar kucuk bir kedi vardi. Kedi her zaman cok fazla guzel cicek yemek istiyordu. Bir gun bir magazaya gitti. Cicekleri gordu ve onu kendisine istedi. Annesinden "Hadi bakalim!" diye bagirdi...

Observation: Sentences are fluent and grammatically correct. Character continuity is partially maintained. Narrative coherence is still limited.

Usage

Install:

pip install torch sentencepiece huggingface_hub

Python:

import torch
import sentencepiece as spm
from huggingface_hub import hf_hub_download
from model import TinyTurkGPTV27, TinyTurkV27Config

REPO = "stunmuffin/TinyTurk-v2.7d-bigstories-384"

ckpt_path = hf_hub_download(REPO, "pytorch_model.bin")
tok_path = hf_hub_download(REPO, "tokenizer_v11_bpe_512.model")

ckpt = torch.load(ckpt_path, map_location="cpu")
config = TinyTurkV27Config(**ckpt["config"])
model = TinyTurkGPTV27(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

sp = spm.SentencePieceProcessor(model_file=tok_path)

prompt = "Bir zamanlar kucuk bir kedi vardi."
ids = [sp.bos_id()] + sp.encode(prompt, out_type=int)
input_ids = torch.tensor([ids])

with torch.no_grad():
    out = model.generate(input_ids, max_new_tokens=300,
                         temperature=0.9, top_k=40)

print(sp.decode(out[0].tolist()))

See example.py for a runnable script.

Limitations

Important: This model has several known limitations:

1. Narrative coherence: The model produces fluent sentences but does not maintain a consistent story arc. Character continuity is partial.
2. Translated training data: The corpus is an automatic translation of TinyStories (English). English proper names (Alice, Tom, Tim) may appear in Turkish output. Some phrases are literal translations.
3. Small vocabulary: 512 BPE subwords.
4. Model size: 1.12M parameters is too small for robust narrative modeling.
5. No instruction tuning: Raw pretrained language model.
6. No safety alignment: Outputs may be incoherent or inappropriate.

Intended Use

• Research on context-length scaling in tiny models
• Turkish sentence-level and short-paragraph generation
• Educational demonstrations
• On-device inference experiments

Out-of-Scope Use

• Long-form storytelling
• Instruction following / chat
• Any safety-critical application

Model Family

Citation

@misc{tinyturk_v27d_bigstories_384_2026,
  title = {TinyTurk v2.7d BigStories-384: Turkish 1.12M-parameter LM with 384-token context},
  author = {TinyTurk Project},
  year = {2026}
}

TinyTurk v2.7d BigStories-384 is an experimental research artifact. Not a product.