squeal_lstm_5m
squeal_lstm_5m is a compact ~5.8M parameter language model pretrained from scratch on Russian-language text, based on LSTM architecture.
This is a base model (pretraining only, without instruction tuning).
Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex text generation or factual tasks will be limited.
Model Description
- Architecture: LSTM decoder with projection layer
- Parameters: ~5.8M
- Tokenizer: Custom Unigram SentencePiece, vocab_size = 4,000, character_coverage = 0.9999
- Context length: 256 tokens
Architecture Details
| Parameter | Value |
|---|---|
| embed_dim | 256 |
| hidden_dim | 512 |
| num_layers | 2 |
| dropout | 0.3 |
| vocab_size | 4,000 |
Training Details
- Dataset: 210,000 documents, ~500MB — opensubtitles (23.8%), habr (14.3%), taiga_proza (14.3%), ru_news (14.3%), cultura_ru_edu (11.9%), fineweb2_ru (11.9%), wikipedia (9.5%)
- Preprocessing: Unicode normalization (NFKC), Cyrillic-ratio filtering, degenerate-repetition filtering, digit/punctuation-ratio filtering, Wikipedia, Fineweb2, cultura paragraph-level chunking, short-utterance grouping, exact deduplication (SHA-256), and approximate deduplication (MinHash/LSH)
- Training Setup: Trained on a Tesla T4 (fp16), 2 epochs, up to step 5,870
- Sequence length: 256 tokens
- Batch size: 128
- Learning rate: 3e-3 (cosine schedule)
Evaluation
| Step | Epoch | Eval Loss | Perplexity |
|---|---|---|---|
| 1000 | 0.34 | 4.408 | 82.2 |
| 2000 | 0.68 | 3.994 | 54.1 |
| 3000 | 1.02 | 3.865 | 47.8 |
| 4000 | 1.36 | 3.800 | 44.7 |
| 5000 | 1.70 | 3.769 | 43.4 |
| 5870 | 2.00 | 3.763 | 43.1 |
Usage
import torch
import torch.nn as nn
from transformers import AutoTokenizer, PreTrainedModel, PretrainedConfig
from transformers.modeling_outputs import CausalLMOutput
class LSTMConfig(PretrainedConfig):
model_type = "lstm_lm"
def __init__(self, vocab_size=4000, embed_dim=256, hidden_dim=512, num_layers=2, dropout=0.3, **kwargs):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.embed_dim = embed_dim
self.hidden_dim = hidden_dim
self.num_layers = num_layers
self.dropout = dropout
class LSTMLanguageModel(PreTrainedModel):
config_class = LSTMConfig
tied_weights_keys = []
@property
def all_tied_weights_keys(self):
return {}
def __init__(self, config):
super().__init__(config)
self.embedding = nn.Embedding(config.vocab_size, config.embed_dim, padding_idx=0)
self.rnn = nn.LSTM(config.embed_dim, config.hidden_dim, num_layers=config.num_layers, batch_first=True, dropout=config.dropout)
self.dropout = nn.Dropout(config.dropout)
self.proj = nn.Linear(config.hidden_dim, config.embed_dim, bias=False)
self.lm_head = nn.Linear(config.embed_dim, config.vocab_size, bias=False)
def forward(self, input_ids, labels=None, **kwargs):
x = self.dropout(self.embedding(input_ids))
out, _ = self.rnn(x)
out = self.dropout(out)
logits = self.lm_head(self.proj(out))
loss = None
if labels is not None:
loss = nn.CrossEntropyLoss(ignore_index=0)(logits.view(-1, logits.size(-1)), labels.view(-1))
return CausalLMOutput(loss=loss, logits=logits)
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_lstm_5m")
model = LSTMLanguageModel.from_pretrained("Squeal-Studio/squeal_lstm_5m")
def generate(prompt, max_new_tokens=200, temperature=0.9, top_k=50, repetition_penalty=1.05):
ids = [tokenizer.bos_token_id] + tokenizer.encode(prompt)
input_ids = torch.tensor([ids])
with torch.no_grad():
for _ in range(max_new_tokens):
logits = model(input_ids).logits[:, -1, :] / temperature
for token_id in set(input_ids[0].tolist()):
logits[0, token_id] /= repetition_penalty
values, _ = torch.topk(logits, top_k)
logits[logits < values[:, -1:]] = -float("inf")
next_id = torch.multinomial(torch.softmax(logits, dim=-1), 1)
if next_id.item() == tokenizer.eos_token_id:
break
input_ids = torch.cat([input_ids, next_id], dim=1)
return tokenizer.decode(input_ids[0].tolist(), skip_special_tokens=True)
print(generate("Однажды в лесу"))
Generation Examples
Prompt: Однажды в лесу
Однажды в лесу я шла только за проживки с маршами! И тогда я была очевидно!
Я отвечаю за нашу природу, где я видел свой фамилии Греции и смоктам все твои
руки гостей повелились, а потом прошли к золотым дяде Весели.
Prompt: Президент заявил
Президент заявил о ситуации на южном строительстве и наступил в суд с юридическим
лицом, переодевшим охрану утопения от украинского "нападения" в проведение акций
протеста, сообщает ТАСС.
Prompt: Наука доказала
Наука доказала, в конце XVIII века в здании, с 1 апреля 1941 по 1952 год.
Всероссийское здание (Войнкот) в город — деревня в Карабахскую сбережь и
присвоено в состав Гракого Сухохенского сельсовета.
Prompt: Linux - это
Linux - это издательная проверка копирования forendispace.ru
Scope & Limitations
- Designed for architectural experiments and educational purposes
- Small vocab (4,000 tokens) limits handling of rare words and emoji
- Context length of 256 tokens limits long-range coherence
- Not intended for production use
License
Apache 2.0
Join us!
Our WebSite: https://squealstudio.ru