AhıskaAI-110M-Experimental-v0.1
AhıskaAI-110M-Experimental-v0.1 is a 110M parameter Small Language Model (SLM) trained from scratch as part of the experimental release series by AhıskaAI. This series focuses on evaluating early-stage convergence and reasoning behavior on synthetic Turkish datasets (covering math, web content, and narrative stories).
Experimental Setup & Scope
- Objective: Test synthetic data density (Math + Web + Stories) and sequence modeling performance on a compact 110M Llama architecture.
- Training Epochs: Pre-trained for 0.3 epochs across a ~3.5 GB synthetic Turkish corpus.
- Context Length: 1024 tokens.
Model Details
- Architecture:
LlamaForCausalLM(Standard MHA) - Parameters: ~110M
- Hidden Size: 768
- Intermediate Size: 2048
- Attention Heads: 12 (Query) / 12 (Key/Value)
- Layers: 12
- Context Length: 1024 tokens
- Vocabulary Size: 32,000
- Precision:
float32/float16
Pre-training Dataset Mixture
The model was pre-trained on a synthetic Turkish corpus (~3.5 GB total):
| Data Source | Approx. Size | Primary Focus | Link |
|---|---|---|---|
| BILGEM AI Synthetic Web | ~2.0 GB | General Web, Knowledge & Grammar | BILGEM-AI/BILGE-Synthetic-Web |
| BILGEM AI Synthetic Math | ~750 MB | Mathematical Reasoning & Logic | BILGEM-AI/BILGE-Synthetic-Math |
| BILGEM AI Synthetic Stories | ~750 MB | Narrative Flow & Text Generation | BILGEM-AI/BILGE-Synthetic-Stories |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AhıskaAI/AhıskaAI-110M-Experimental-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
device_map="auto"
)
prompt = "Bir zamanlar uzak bir ülkede"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.7,
top_p=0.9,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
About AhıskaAI
AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.