TinyBrainBot-100M-v3-Instruct
A 100M-parameter chat/instruct model that beats SupraLabs/Supra2-100M-Instruct on 6/7 benchmarks on the official EleutherAI LM-Eval Harness — and actually converses.
Built by SFT from tinybrainbot-100m-v3-base (chat / instruction-following / recall data), then a WiSE-FT weight soup (α=0.7 interpolation back toward the base) to recover benchmark accuracy while keeping chat.
- Architecture: Llama-compatible, 100.1M params (768/12L/12h·4kv, ctx 1024, vocab 32k).
- Chat template:
<|user|>\n{msg}\n<|end|>\n<|assistant|>\n
Benchmarks (EleutherAI lm-eval, 0-shot, acc_norm; WinoGrande/MMLU = acc)
| Benchmark | This model | Supra2-100M-Instruct | Δ |
|---|---|---|---|
| ARC-Easy | 54.5 | 44.4 | +10.1 |
| ARC-Easy (raw acc) | 58.3 | 51.4 | +6.9 |
| ARC-Challenge | 29.2 | 24.7 | +4.5 |
| ARC-Challenge (raw acc) | 26.7 | 22.7 | +4.0 |
| OpenBookQA | 32.8 | 30.4 | +2.4 |
| OpenBookQA (raw acc) | 21.6 | 17.2 | +4.4 |
| PIQA | 65.5 | 64.4 | +1.1 |
| PIQA (raw acc) | 65.3 | 64.3 | +1.0 |
| WinoGrande | 51.4 | 50.5 | +0.9 |
| MMLU | 26.1 | 25.8 | +0.3 |
| HellaSwag | 32.8 | 35.9 | −3.1 |
| HellaSwag (raw acc) | 30.1 | 31.5 | −1.4 |
6/7 vs Supra2-Instruct — only HellaSwag lost.
Reproduce these numbers
EleutherAI lm-eval-harness v0.4.12, 0-shot, on the HF repo (not the GGUF — llama.cpp's --multiple-choice path under-reports these tasks):
lm_eval --model hf \
--model_args pretrained=nkthebass/tinybrainbot-100m-v3-instruct,dtype=float32 \
--tasks hellaswag,arc_easy,arc_challenge,openbookqa,winogrande,piqa,mmlu \
--num_fewshot 0 --batch_size 32
Metrics: acc_norm for HellaSwag / ARC / OpenBookQA / PIQA; acc for WinoGrande & MMLU.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-100m-v3-instruct")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-100m-v3-instruct")
prompt = "<|user|>\nWhat is the capital of France?\n<|end|>\n<|assistant|>\n"
ids = tok(prompt, return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=40)[0][ids.shape[1]:], skip_special_tokens=True))
# -> "Paris."
GGUF
An F16 GGUF is included (tinybrainbot-100m-v3-instruct-f16.gguf) for llama.cpp / Ollama / LM Studio, with the add_space_prefix=false + leading-space chat template baked in so it chats faithfully out of the box.
Limitations
Coherent for short chat and 1–2 sentence answers; longer generations can confabulate (the 100M ceiling on factual generation). Arithmetic is unreliable — use tinybrainbot-100m-v3-math. Not safety-tuned.