squeal_ai_8m-instruct
squeal_ai_8m-instruct is a compact ~8M parameter language model, fine-tuned for instruction following on Russian-language data. It belongs to the squeal_ai_ model family by Squeal Studio.
This is the instruct version of squeal_ai_8m-base, fine-tuned with Supervised Fine-Tuning (SFT) on an instruction dataset. No RLHF or DPO alignment stage was applied.
Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex instructions or factual tasks will be limited.
Model Description
- Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
- Parameters: ~8M
- Tokenizer: Custom BPE, vocab_size = 24,000
- Context length: 1,536 tokens
- Base model: squeal_ai_8m-base
- Fine-tuning method: Supervised Fine-Tuning (SFT) only, no RLHF/DPO
Architecture Details
| Parameter | Value |
|---|---|
| hidden_size | 192 |
| num_hidden_layers | 8 |
| num_attention_heads | 6 |
| num_key_value_heads | 3 |
| intermediate_size | 512 |
| max_position_embeddings | 1536 |
| vocab_size | 24,000 |
Training Details
- Base model: squeal_ai_8m-base, pretrained on Russian Wikipedia + OpenSubtitles (RU)
- SFT dataset: publicly available Russian-language instruction dataset from Hugging Face
- Training setup: Trained on a Tesla T4 (fp16) up to step 1400
- Chat template: included (chat_template.jinja) for use with tokenizer.apply_chat_template
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_8m-instruct")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_8m-instruct")
prompt = "<|user|>\nКак поднять настроение?\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Scope & Limitations
- Very small parameter count means limited factual knowledge and reasoning ability
- SFT only — no RLHF or DPO, so alignment and instruction-following robustness are limited compared to larger aligned models
- May produce repetitive, incoherent, or factually incorrect responses
- Not intended for production, safety-critical, or high-stakes use cases
- Best suited for architectural experiments, baseline comparisons, and educational purposes
License
Apache 2.0