scribe
A ~30M-parameter small language model built from scratch in PyTorch and trained on
TinyStories. It's a decoder-only GPT —
hand-written multi-head causal attention, pre-LayerNorm blocks, GELU MLP, tied LM head — with no
transformers/nanoGPT modelling code. The architecture is deliberately GPT-2-compatible, so
these weights load directly into ember, a from-scratch
LLM inference server.
📦 Code, training, and full write-up (incl. the maths): https://github.com/mbsdeepak/scribe
What it does
Give it a prompt and it continues it as a simple, coherent children's story:
"Once upon a time" → there was a little girl named Lily. She loved to read magazines. One day, she found a magazine in her mom's room... "Mommy, look at that magazine!" Lily said... Her mom looked at the magazine and smiled. "That's very nice, Lily. But be careful with it, it might break."
"One day, a boy named Deepak" → went to the park every day. He saw a large tree and wanted to climb it. He started to climb it, but it was too heavy for him. He tried to jump, but he fell.
Model
| Architecture | decoder-only transformer (GPT-2 family) |
| Parameters | ~30.0M (19.3M of it the tied 50257-token embedding) |
| Layers / heads / width | 6 / 6 / 384 |
| Context length | 256 tokens |
| Tokenizer | GPT-2 BPE (tiktoken) |
| Training | TinyStories, AdamW + warmup/cosine, ~3.4h on Apple-Silicon MPS |
| Validation loss | 1.74 (cross-entropy) |
Files
model.safetensors— the weights (fp32)config.json— architecture config to rebuild the model
Usage
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import json, torch
# get the code: git clone https://github.com/mbsdeepak/scribe
from config import GPTConfig
from src.model import GPT
from src.tokenizer import Tokenizer
cfg_json = json.load(open(hf_hub_download("mbsdeepak/scribe", "config.json")))
gc = GPTConfig(n_layer=cfg_json["n_layer"], n_head=cfg_json["n_head"],
n_embd=cfg_json["n_embd"], block_size=cfg_json["max_position"])
model = GPT(gc).eval()
model.load_state_dict(load_file(hf_hub_download("mbsdeepak/scribe", "model.safetensors")))
tok = Tokenizer()
ids = torch.tensor([tok.encode("Once upon a time")])
out = model.generate(ids, max_new_tokens=120, temperature=0.8, top_k=200)[0].tolist()
print(tok.decode(out))
Or serve it with ember for a streaming, OpenAI-compatible API.
Limitations
A 30M model on TinyStories writes simple, coherent children's-story English — not general-purpose text. It's a learning/portfolio project: the goal is understanding an LLM end-to-end (architecture, training, serving), not competing with production models. Context is 256 tokens; architecture is vanilla GPT-2 (no RoPE/RMSNorm/SwiGLU).
License
MIT