Dasha 215M (chat)
A decoder-only transformer trained from scratch (not fine-tuned from an existing pretrained LLM). This is the instruction/chat-tuned (SFT) version.
This is an early checkpoint: pretrained on a relatively small, general (Wikipedia ru+en) corpus. A larger, much more densely-trained successor (higher tokens/parameter ratio, broader web+math data, same architecture) is currently in progress -- this card will be updated when it replaces this checkpoint.
Architecture
- RMSNorm (pre-norm), RoPE, Grouped-Query Attention (GQA: 16 query heads / 4 KV heads), QK-Norm, SwiGLU MLP, tied embeddings
- 213.9M parameters, 16 layers, dim=1024, context length 2048
- torch SDPA attention with an automatic flash-attn fallback on supported architectures (Ampere/Ada/Hopper)
Training
- Optimizer: hybrid Muon (2D hidden-layer matrices) + Adam-mini (embeddings) + AdamW (norm gains)
- Pretrain data: Wikipedia ru+en
- SFT: ru/en multi-turn dialogue (Vikhrmodels/GrandMaster-PRO-MAX), ChatML-style format
Evaluation
Evaluated with lm-evaluation-harness
(0-shot, acc) and the official MERA benchmark for Russian.
Numbers are honest, not cherry-picked -- at 214M parameters and this data scale,
the model is a fluent, grammatical talker with weak multi-step reasoning and
weak math, which is the expected profile for this size class, not a bug.
English common-sense/reasoning (vs. Pythia-160M, a similarly-sized public baseline):
| Task | Dasha 215M | Pythia-160M |
|---|---|---|
| PIQA | 55.4% | 59.1% |
| ARC-Easy | 26.4% | 37.6% |
| WinoGrande | 49.1% | 50.5% |
| BoolQ | 38.2% | 44.1% |
| HellaSwag | 26.8% | 28.3% |
Russian (XWinograd-ru, XStoryCloze-ru, XNLI-ru, Belebele-ru):
| Task | Dasha 215M |
|---|---|
| XWinograd-ru | 52.4% |
| XStoryCloze-ru | 51.8% |
| XNLI-ru | 35.2% |
| Belebele-ru | 23.0% |
MERA (aggregate, official submission): 0.195 -- near-random overall, with math/code tasks essentially at 0 (as expected: no dedicated math/code data in this checkpoint's pretrain mix). The in-progress successor targets this specifically with a denser, more diverse pretrain corpus.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("dariumi/dasha-215m")
model = AutoModelForCausalLM.from_pretrained("dariumi/dasha-215m", trust_remote_code=True, dtype=torch.bfloat16)
ids = tok("Привет!", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=50, do_sample=True, temperature=0.8)
print(tok.decode(out[0]))
Note: generate() via this HF wrapper reprocesses the full sequence each
step (no KV-cache), so it is correct but not fast for long generations.