AhiskaAI-134m-Base-v0.2
AhiskaAI-134m-Base-v0.2 is a 134 million parameter Small Language Model (SLM) built from scratch. It represents the second generation of the AhiskaAI organization's mission to develop efficient, high-quality, native Turkish language models.
Model Details
- Architecture: Llama-based architecture.
- Parameters: 134M.
- Context Window: 1024 tokens.
- Tokenizer: Custom BPE Tokenizer (Vocabulary Size: 32,000).
- Training Framework: PyTorch & Transformers.
Data Curation (The "Quality over Quantity" Approach)
A major leap in the v0.2 release is the transition to data-centric AI.
- Raw Data (v0.1): 5GB of raw corpus.
- Curated Data (v0.2): 1.2GB of meticulously filtered, high-quality Turkish text.
- Process: Approximately 75% of low-quality, noisy, and redundant data was eliminated to ensure the model focuses on grammatical correctness and linguistic density.
Key Improvements from v0.1
- Architecture Shift: Migrated from standard GPT-2 to a modern Llama-based architecture for superior sequence modeling.
- Normalization: Implemented RMSNorm for training stability.
- Positional Encoding: Adopted RoPE (Rotary Positional Embeddings).
- Activation: Switched to SiLU activation function.
- Precision: Trained in bfloat16 for efficiency on consumer-grade hardware.
Training Logs
The graph above demonstrates the training convergence of AhiskaAI-134m-Base-v0.2. The stable decline in loss confirms the effective alignment of the model architecture with the curated Turkish dataset.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("AhiskaAI/AhiskaAI-134m-Base-v0.2")
tokenizer = AutoTokenizer.from_pretrained("AhiskaAI/AhiskaAI-134m-Base-v0.2")
text = "Türkiye Cumhuriyeti"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Hardware
Trained on NVIDIA RTX 4050 6GB Laptop GPU.
Future Plans
Instruct Version: AhiskaAI-134m-IT-v0.2 (Fine-tuned for chat and instructions).
Preference Alignment: v0.2.1 DPO (Direct Preference Optimization) is currently under development to refine response quality.
About AhiskaAI
AhiskaAI is an independent initiative dedicated to pushing the boundaries of Small Language Models in the Turkish language. Follow us on Hugging Face for updates and new releases.