squeal_ai_20m-base
squeal_ai_20m-base is a compact ~20M parameter language model pretrained from scratch on Russian-language text. It belongs to the squeal_ai_ model family by Squeal Studio.
This is a base model (pretraining only, without instruction tuning).
Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex text generation or factual tasks will be limited.
Model Description
- Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
- Parameters: ~20M
- Tokenizer: Custom BPE, vocab_size = 24,000
- Context length: 1,024 tokens
Architecture Details
| Parameter | Value |
|---|---|
| hidden_size | 352 |
| num_hidden_layers | 8 |
| num_attention_heads | 8 |
| num_key_value_heads | 4 |
| intermediate_size | 1024 |
| max_position_embeddings | 1024 |
| vocab_size | 24,000 |
Training Details
- Dataset: cultura_ru_edu (Russian fineweb-edu analog, 65.3%), fineweb2_ru (20.6%), and Russian Wikipedia (14.2%) — 1,509,932 documents, ~4.7gb, ~2.69B characters, estimated ~700M–1B tokens.
- Preprocessing: Unicode normalization (NFKC), Cyrillic-ratio filtering, degenerate-repetition filtering, digit/punctuation-ratio filtering, Wikipedia paragraph-level chunking, short-utterance grouping, exact deduplication (SHA-256), and approximate deduplication (MinHash/LSH)
- Training Setup: Trained on a Tesla T4 (fp16) up to step 3,200
Evaluation
| Step | Epoch | Eval Loss | Perplexity |
|---|---|---|---|
| 3200 | 0.7485 | 4.057 | 57.85 |
Eval loss had largely plateaued by this point in training.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_20m-base")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_20m-base")
inputs = tokenizer("Привет, как дела?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Scope & Limitations
- Designed primarily for architectural testing, baseline experiments, and lightweight tasks
- Lacks instruction tuning and high parameter capacity required for multi-turn conversational AI or high-precision factual recall
- Not intended for production environments
License
Apache 2.0
Join us!
Our WebSite: https://squealstudio.ru