mara-small
A 24.6M parameter GPT trained from scratch (random init → coherent children's stories) on the TinyStories corpus. Designed as the base model of the mara project: a small-model-first family targeting browser inference and microcontrollers.
Live demo (runs in your browser, no server): https://huggingface.co/spaces/jaswanthsanjay88/mara-demo
Architecture
| Config | Value |
|---|---|
| Parameters | 24,648,192 |
| Layers | 6 |
| d_model | 512 |
| Heads | 8 (head_dim 64) |
| FFN | SwiGLU, hidden 1,536 |
| Positional | RoPE |
| Norm | RMSNorm (pre-norm) |
| Context | 512 tokens |
| Vocab | 8,192 custom byte-level BPE (trained on TinyStories) |
Training
- Data: TinyStories (~469M train tokens, packed uint16 stream)
- Steps: 8,000 · batch 64 × seq 512 = 32,768 tok/step ≈ 262M tokens seen
- Optimizer: AdamW (fused), β=(0.9, 0.95), wd 0.1 on ≥2D params
- Schedule: cosine 6e-4 → 6e-5, 200 warmup steps, grad clip 1.0
- Precision: fp16 autocast (T4)
- Final training loss: ~1.40
Files
| File | Size | Purpose |
|---|---|---|
model.safetensors | ~50 MB | fp16 PyTorch weights (training/fine-tuning) |
model_fp16.onnx | 50 MB | browser inference via WebGPU |
model_int8.onnx | 27 MB | browser inference via WASM (all browsers) |
tokenizer.json | — | custom 8k BPE (`< |
config.json | — | architecture hyperparameters |
Usage (PyTorch)
The architecture is custom (mara/model.py in the source repo): RoPE + SwiGLU + tied embeddings. Load weights like so:
import json, torch
from safetensors.torch import load_file
from mara.model import Mara, MaraConfig # from the mara repo
from mara.tokenizer import load_tokenizer
cfg = MaraConfig(**json.load(open("config.json")))
model = Mara(cfg)
model.load_state_dict({k: v.float() for k, v in load_file("model.safetensors").items()})
model.eval()
tok = load_tokenizer("tokenizer.json")
ids = torch.tensor([tok.encode("Once upon a time")])
print(tok.decode(model.generate(ids, max_new_tokens=200)[0].tolist()))
Limitations
- Children's-story English only — out-of-domain prompts degrade quickly
- Occasional pronoun/gender confusion, typical at this scale
- No instruction-following or world knowledge; it is a pure next-token storyteller
- Context window of 512 tokens
Roadmap
This model is the teacher for a distilled intent-parsing student that runs fully offline on an ESP32-S3 for smart-home control.