squeal_ai_20m-instruct
squeal_ai_20m-instruct is a compact ~20M parameter language model, fine-tuned for instruction following on Russian-language data. It belongs to the squeal_ai_ model family by Squeal Studio.
This is the instruct version of squeal_ai_20m-base, fine-tuned with Supervised Fine-Tuning (SFT) on an instruction dataset. No RLHF or DPO alignment stage was applied.
Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex instructions or factual tasks will be limited.
Model Description
- Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
- Parameters: ~20M
- Tokenizer: Custom BPE, vocab_size = 24,000
- Context length: 1,024 tokens
- Base model: squeal_ai_20m-base
- Fine-tuning method: Supervised Fine-Tuning (SFT) only, no RLHF/DPO
Architecture Details
| Parameter | Value |
|---|---|
| hidden_size | 352 |
| num_hidden_layers | 8 |
| num_attention_heads | 8 |
| num_key_value_heads | 4 |
| intermediate_size | 1024 |
| max_position_embeddings | 1024 |
| vocab_size | 24,000 |
Training Details
- Base model: squeal_ai_20m-base, pretrained on cultura_ru_edu, fineweb2_ru, and Russian Wikipedia
- SFT dataset: This model was fine-tuned using ru_turbo_alpaca, created by IlyaGusev
- Dataset License: CC-BY 4.0
- Modifications: Added examples of self-identification and reformatted to a plain
User:/Bot:prompt structure
- Training setup: Trained on a Tesla T4 (fp16) up to step 2,400
- Chat template: none — no
chat_template.jinjais included; prompts must be manually formatted as shown below
Usage
This model does not use tokenizer.apply_chat_template. Prompts must follow the plain-text format below:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_20m-instruct")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_20m-instruct")
prompt = "User: Как поднять настроение?\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Scope & Limitations
- Very small parameter count means limited factual knowledge and reasoning ability
- SFT only, no RLHF or DPO, so alignment and instruction-following robustness are limited compared to larger aligned models
- No chat template is provided, the model expects the exact
User: ... \nBot:format shown above and may behave unpredictably with other prompt structures - May produce repetitive, incoherent, or factually incorrect responses
- Not intended for production, safety-critical, or high-stakes use cases
- Best suited for architectural experiments, baseline comparisons, and educational purposes
License
Apache 2.0
Join us!
Our WebSite: https://squealstudio.ru