TinyBrainBot-25M-v4-Base
A 25M-parameter language model trained from scratch on 11B tokens, part of the TinyBrainBot v4 series. 25.2M parameters in the transformer itself, plus a 5.4M tied embedding for its 14k vocabulary (30.6M total). It is a base model: it continues text.
It beats Pythia-70M, which has more than twice the parameters and saw 300B tokens, on 10 of 13 benchmarks (ties a further one), and averages 38.5 against Pythia-70M's 35.9.
| Benchmark (0-shot) | TinyBrainBot-25M-v4 | Pythia-31M | Pythia-70M |
|---|---|---|---|
| ARC-Easy | 42.7 | 33.7 | 36.0 |
| ARC-Easy (raw acc) | 47.3 | 37.2 | 38.2 |
| ARC-Challenge | 24.9 | 21.0 | 21.8 |
| ARC-Challenge (raw acc) | 21.0 | 16.6 | 17.6 |
| SciQ | 65.9 | 51.5 | 56.4 |
| SciQ (raw acc) | 73.7 | 60.0 | 64.0 |
| OpenBookQA | 31.2 | 26.8 | 25.4 |
| OpenBookQA (raw acc) | 17.2 | 13.0 | 12.6 |
| MMLU | 23.7 | 22.9 | 22.9 |
| HellaSwag | 37.2 | 33.6 | 33.8 |
| HellaSwag (raw acc) | 32.5 | 31.2 | 30.8 |
| PIQA | 60.2 | 56.6 | 59.2 |
| PIQA (raw acc) | 60.0 | 57.5 | 59.8 |
| WinoGrande | 52.6 | 49.3 | 52.9 |
| Social IQa | 37.2 | 35.1 | 35.4 |
| CommonsenseQA | 20.3 | 19.7 | 19.8 |
| LAMBADA (OpenAI) | 23.4 | 17.8 | 23.4 |
| BoolQ | 58.8 | 51.0 | 59.6 |
| MathQA | 21.9 | 19.5 | 20.4 |
| MathQA (raw acc) | 23.2 | 19.2 | 20.9 |
| Average | 38.5 | 33.7 | 35.9 |
EleutherAI lm-eval-harness v0.4.13, 0-shot, up to 2,000 examples per task, acc_norm where
the task reports it and acc otherwise. All three models were run by us on the same harness
and settings.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-25m-v4-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
ids = tok("Once upon a time, a little fox", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80) # defaults from generation_config.json
print(tok.decode(out[0], skip_special_tokens=True))
GGUF (llama.cpp, LM Studio, Ollama): tinybrainbot-25m-v4-base-F16.gguf (59 MB) and
tinybrainbot-25m-v4-base-Q8_0.gguf (31 MB) are in this repo. They tokenize and generate
identically to the transformers version.
llama-cli -m tinybrainbot-25m-v4-base-Q8_0.gguf -p "Once upon a time, a little fox" -n 80 --temp 0.7 --repeat-penalty 1.15
Default sampling: temperature 0.7, top_p 0.95, top_k 40, repetition_penalty 1.15.
Samples
Unedited, at the default settings.
Once upon a time, a little fox named Mia decided to start her own small business selling handmade crafts at the local community center. She wanted to make sure that everyone had enough supplies and opportunities to buy their favorite items. So, she gathered some friends and helped them find a creative idea for making their own goods.
Dear diary, today was a fun and exciting day with lots of memories! I found out that every single day, we learned about the fascinating world of animal behavior and how it affects us all. So let's continue exploring this amazing subject together!
Photosynthesis is the process by which light energy is converted into chemical energy (e.g., glycolysis). Photosynthetic bacteria convert sunlight, carbon dioxide, and water to glucose (a sugar-sweetened food), oxygen, and nitrogen, respectively.
Model
| Architecture | Llama-compatible decoder, RoPE, SwiGLU, RMSNorm, grouped-query attention |
| Shape | hidden 384, 16 layers, 6 heads of 64, 2 KV heads, FFN 1024 |
| Parameters | 25.18M non-embedding + 5.38M tied embedding = 30.55M |
| Context | 1,024 tokens |
| Vocabulary | 14,000 |
Deep and narrow on purpose: at this size, adding layers helps more than adding width.
Tokenizer
A new 14k byte-level BPE trained for this model on a sample of its own training mix:
- Newlines and indentation survive exactly, so code, lists and verse keep their shape.
- Digits are split one per token. "1234" and "1243" share their pieces, which is what small models need to learn arithmetic.
- No unknown tokens. Every byte is in the base alphabet.
- Chat and reasoning markers (
<|user|>,<|assistant|>,<|end|>,<think>,</think>) are single tokens, reserved for fine-tuning.<|endoftext|>(id 0) separates documents.
Training
11.0B tokens on 2x Tesla V100 16GB, about 22 hours, fp16, AdamW (0.9, 0.95), weight decay 0.1, peak LR 2e-3, 131k tokens per step, warmup-stable-decay schedule.
Stage 1 - 9.7B tokens, general mix. Every source under 3 epochs.
| Group | Share | Sources |
|---|---|---|
| Web | 37% | FineWeb-Edu 28, DCLM 9 |
| Reference | 13.5% | Wikipedia article leads 12, full Wikipedia 1.5 |
| Narrative | 11% | Project Gutenberg 8, BookCorpusOpen 3 |
| Code | 9% | The Stack (smol) 7.5, Python/JS 1.5 |
| Math | 9% | MegaMath-Web-Pro 6, FineMath-4+ 3 |
| Synthetic textbooks | 7.5% | Cosmopedia (Stanford, OpenStax, auto-math) |
| Q&A and facts | 7% | distilled Q&A, short facts, cause-and-effect explanations |
| Educational PDFs | 5.5% | FinePDFs-Edu |
| Reasoning | 0.5% | distilled short reasoning |
Stage 2 - 500M tokens. The same mix with 7.8% added: everyday physical-commonsense writing (2.5%), procedural "goal and method" writing (1.8%), worked order-of-operations problems (2%) and short facts (+1.5%). All of it is generated text written for this purpose; none of it is benchmark data. Stage 1 ended with a short LR decay, which stage 2 re-warmed.
Stage 3 - 803M token anneal. LR decays to zero over a fact-dense mix: Cosmopedia, FineMath, FinePDFs-Edu, Wikipedia leads, distilled Q&A and reasoning, plus the stage 2 additions.
The packed corpus was block-shuffled and its document density checked shard by shard before training, so every stage is an even mix rather than one source after another.
Reproduce the benchmarks
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-25m-v4-base,dtype=float32 \
--tasks arc_easy,arc_challenge,sciq,openbookqa,mmlu,hellaswag,piqa,winogrande,social_iqa,commonsense_qa,lambada_openai,boolq,mathqa \
--num_fewshot 0 --limit 2000 --batch_size 8 --seed 1234
TinyBrainBot v4
- TinyBrainBot-350M-v4-Thinking
- TinyBrainBot-100M-v4-Thinking
- TinyBrainBot-25M-v4-Base (this model)