AhıskaAI-135M-Base-v0.3
AhıskaAI-135M-Base-v0.3 is a ~135 million parameter Small Language Model (SLM) trained completely from scratch. Building upon the learnings of the v0.2 release, v0.3 introduces architectural optimizations (such as Grouped-Query Attention), an expanded and multi-domain pre-training corpus, and enhanced training efficiency using Liger Kernel.
Key Improvements from v0.2 to v0.3
- Architectural Evolution (GQA Support): Transitioned to Grouped-Query Attention (GQA) with 12 query heads and 4 key-value heads (
num_key_value_heads: 4). This significantly reduces KV-cache memory overhead and boosts inference throughput compared to standard MHA in v0.2. - Diversified Pre-training Mixture: Expanded beyond single-source corpora to a multi-domain dataset totaling ~2.04 GB across Turkish Wikipedia (%53.2), Turkish Stories (%21.6), Python Code (%17.6), and English FineWeb (%7.6).
- Training Acceleration & Efficiency: Utilized Liger Kernel integration (
apply_liger_kernel_to_llama) andadamw_torch_fusedoptimization inbfloat16precision, maximizing hardware utilization. - Hardware & Scale: Scaled pre-training across dual Kaggle NVIDIA T4 GPUs for 11 hours, overcoming consumer laptop VRAM limitations while preserving tight parameter footprints.
Model Details
- Architecture:
LlamaForCausalLMwith GQA - Parameters: ~135M
- Hidden Size: 768
- Intermediate Size: 3072
- Attention Heads: 12 (Query) / 4 (Key/Value)
- Layers: 14
- Context Length: 512 tokens
- Vocabulary Size: 32,000
- Precision:
float16
Pre-training Dataset Distribution
The model was pre-trained on a balanced multi-domain dataset (~2.04 GB total):
| Data Source | Share (%) | Size | Primary Domain |
|---|---|---|---|
| BILGEM AI Turkish Wikipedia | 53.2% | 1.09 GB | General Knowledge & Turkish Grammar |
| BİLGEM Turkish Stories | 21.6% | 443 MB | Narrative & Fluency |
| Python Code (The Stack) | 17.6% | 360 MB | Reasoning & Logic |
| FineWeb English | 7.6% | 150 MB | Cross-lingual & General Alignment |
480M Tokens
Training Hardware & Setup
- Hardware: 2x NVIDIA T4 GPUs (Kaggle)
- Training Time: ~11 Hours
- Optimization: SDPA (Scaled Dot-Product Attention), AdamW Fused, Liger Kernel
- Precision:
float16
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AhiskaAI/AhiskaAI-135M-Base-v0.3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
device_map="auto"
)
prompt = "Türkiye Cumhuriyeti'nin başkenti"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.7,
top_p=0.9,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Related Models
- Instruct Version: AhiskaAI/AhiskaAI-135M-IT-v0.3 (SFT ChatML model fine-tuned for instruction following)
About AhıskaAI
AhıskaAI is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.