AhıskaAI-10M-Experimental-v0.1
AhıskaAI-10M-Experimental-v0.1 is an ultra-compact ~10 million parameter Small Language Model (SLM) trained from scratch. As part of the AhıskaAI experimental series, this micro-model is designed to test the absolute lower limits of Turkish language representations, syntax learning, and synthetic task understanding at minimal computational costs.
Experimental Scope & Setup
- Objective: Evaluate how much linguistic structure and reasoning a 10M parameter micro-model can capture when trained on dense synthetic data.
- Training Epochs: Pre-trained for 1 full epoch across a ~3.5 GB synthetic Turkish corpus.
- Vocabulary Efficiency: Uses an optimized 8,000 token vocabulary specifically suited for micro-scale models.
Model Details
- Architecture:
LlamaForCausalLM(Micro MHA) - Parameters: ~10M
- Hidden Size: 288
- Intermediate Size: 768
- Attention Heads: 8 (Query) / 8 (Key/Value)
- Layers: 8
- Context Length: 512 tokens
- Vocabulary Size: 8,000
- Precision:
float32
Pre-training Dataset Mixture
The model was pre-trained on a synthetic Turkish corpus (~3.5 GB total):
| Data Source | Approx. Size | Primary Focus | Link |
|---|---|---|---|
| BILGEM AI Synthetic Web | ~2.0 GB | General Web, Knowledge & Grammar | BILGEM-AI/BILGE-Synthetic-Web |
| BILGEM AI Synthetic Math | ~750 MB | Mathematical Reasoning & Logic | BILGEM-AI/BILGE-Synthetic-Math |
| BILGEM AI Synthetic Stories | ~750 MB | Narrative Flow & Text Generation | BILGEM-AI/BILGE-Synthetic-Stories |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AhıskaAI/AhıskaAI-10M-Experimental-v0.1-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
device_map="auto"
)
prompt = "Türkiye"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=50,
temperature=0.7,
top_p=0.9,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
About AhıskaAI
AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.