TinyBrainBot 303M — Base
A 303M-parameter, LLaMA-architecture language model trained from scratch on a home server (2× NVIDIA Tesla P100). This is the base (pretrained) model — it has not been instruction-tuned, so it completes text rather than following chat instructions.
👉 For the chat/assistant version, see TinyBrainBot 303M Instruct.
Model details
| Parameters | ~303M |
| Architecture | LLaMA-style decoder (RoPE, RMSNorm, SwiGLU, pre-norm) |
| Layers | 24 |
| Hidden size | 1024 |
| Attention heads | 16 (GQA, 4 KV heads) |
| Head dim | 64 |
| FFN size | 2816 |
| Vocab | 32,000 (custom SentencePiece BPE) |
| Context length | 1024 |
| Tied embeddings | Yes |
| Precision | fp16 |
It re-expresses cleanly as a LlamaForCausalLM, so it loads with standard 🤗 Transformers.
Training
Pretrained on a mix of open English text plus synthetic distillation data:
- Web / educational text: FineWeb-Edu, English Wikipedia, OpenWebText2
- Narrative: TinyStories
- Math/reasoning: Orca-Math
- Synthetic fact distillation: Q&A and short-fact data generated by a stronger teacher model (Ling 2.6 Flash) to densify factual coverage.
Total training tokens are on the order of a few billion — far fewer than models like SmolLM/Pythia (hundreds of billions). This is a hobbyist/research model; capability scales with tokens, and this one is intentionally token-limited. (4.7B)
Intended use
- A starting point for continued pretraining or fine-tuning at small scale.
- Text completion and research on small language models.
- Not for chat/instruction following — use the Instruct version for that.
Limitations
- Shallow, fragile factual knowledge. It knows common facts but is sensitive to phrasing and capitalization (e.g. "France" vs "france" can flip the answer).
- Weak at reasoning and math; roughly random on knowledge benchmarks like MMLU (typical for this size).
- Will produce fluent plausible-but-wrong text confidently. Pair with retrieval (RAG) for anything factual.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("your-username/tinybrainbot-303m-base")
model = AutoModelForCausalLM.from_pretrained("your-username/tinybrainbot-303m-base", torch_dtype=torch.float16)
ids = tok("The water cycle is the process by which", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
License
Apache-2.0 (change if you prefer a different license).