squeal_ai_8m-base
squeal_ai_8m-base is a compact ~8M parameter language model pretrained from scratch on Russian-language text. It belongs to the squeal_ai_ model family by Squeal Studio.
This is a base model (pretraining only, without instruction tuning).
Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex text generation or factual tasks will be limited.
Model Description
- Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
- Parameters: ~8M
- Tokenizer: Custom BPE, vocab_size = 24,000
- Context length: 1,536 tokens
Architecture Details
| Parameter | Value |
|---|---|
| hidden_size | 192 |
| num_hidden_layers | 8 |
| num_attention_heads | 6 |
| num_key_value_heads | 3 |
| intermediate_size | 512 |
| max_position_embeddings | 1536 |
| vocab_size | 24,000 |
Training Details
- Dataset: Selected Russian Wikipedia articles and OpenSubtitles (RU) dataset (~350MB raw text, estimated ~50–70M tokens)
- Preprocessing: NFKC normalization, character-ratio filtering, paragraph/document grouping, and MinHash/LSH deduplication
- Training Setup: Trained on a Tesla T4 (fp16) up to step 996
Evaluation
Training and validation loss across pretraining epochs:
| Epoch | Eval Loss | Perplexity |
|---|---|---|
| 1 | 6.567 | 711.08 |
| 2 | 5.999 | 403.03 |
| 3 (final) | 5.688 | 295.20 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_8m-base")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_8m-base")
inputs = tokenizer("Привет, как дела?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Scope & Limitations
- Designed primarily for architectural testing, baseline experiments, and lightweight tasks
- Lacks instruction tuning and high parameter capacity required for multi-turn conversational AI or high-precision factual recall
- Not intended for production environments
License
Apache 2.0