kitchenbot-base
A ~6.85M GPT-2-style language model trained from scratch on cooking recipes over a weekend — part of a hands-on experiment to learn the full pretrain → chat-SFT loop on a single rented GPU.
| Chat fine-tune | bychwa/kitchenbot-chat (LoRA adapter) |
| Training code | github.com/bychwa/kitchenbot |
| Logs | wandb · kitchenbot |
Motivation
I wanted a real end-to-end run I could finish in a weekend: niche data, custom tokenizer, pretrain a small causal LM, then LoRA-tune it for Q&A. Keeping the model tiny was intentional — fit the whole loop on one RTX 3090 pod, focus on process, and ship working artifacts.
Model details
| Architecture | GPT2LMHeadModel |
| Parameters | ~6.85M |
| Layers / emb / heads | 6 / 256 / 8 |
| Context | 256 tokens |
| Vocab | 8000 (ByteLevel BPE, trained on the recipe corpus) |
| Special tokens | <|pad|>, <|unk|>, <|bos|>, <|eos|>, <|user|>, <|assistant|> |
Weights are ~26 MB (safetensors).
Training data
Source: idoyaaran/mise-recipes (streamed from the Hub).
Each example was formatted roughly as:
<|bos|>{title}
Ingredients: {ingredient names}
Steps: {joined steps}<|eos|>
The weekend run used ~10k recipes (MAX_SAMPLES=10000 in the training script).
Hardware (RunPod)
| Spec | Value |
|---|---|
| GPU | 1× NVIDIA GeForce RTX 3090 (24 GB) |
| CUDA | 13.0 |
| Host | Linux (RunPod container) |
| Python | 3.12 |
| Stack | PyTorch 2.5.1+cu121, Transformers 5.14, Datasets, Tokenizers, Accelerate, W&B |
Approximate cost: a few dollars at ~$0.25–0.40/hr for a short pretrain + SFT session.
Training procedure
Causal language modeling (next-token prediction) with Hugging Face Trainer.
| Hyperparameter | Value |
|---|---|
| Learning rate | 3e-4 |
| Warmup steps | 100 |
| Batch size | 16 |
| Grad accumulation | 4 (effective batch 64) |
| Epochs | 1 |
| Max length | 256 |
| Precision | fp16 |
| Optimizer | AdamW |
| LR schedule | linear |
| Seed | 42 |
Results (train)
| Metric | Value |
|---|---|
| Steps | 157 |
| Train runtime | ~24 s (this config on 3090) |
train_loss | ≈ 5.88 |
| Last logged step loss | ≈ 4.19 |
These are training metrics, not a held-out perplexity suite. Loss trended down; the model learns recipe-ish continuations, not general knowledge.
Intended use
- Recipe-style text continuation in the cooking domain
- Base weights for the LoRA chat adapter
kitchenbot-chat - Teaching / portfolio example of a from-scratch SLM pipeline
Not intended as a general assistant, medical/nutrition advice source, or production kitchen system.
Limitations
- Tiny capacity → invents ingredients/steps and mixes dishes
- 256-token context truncates long recipes
- English cooking text bias from the source dataset
- No safety / factuality filtering
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "bychwa/kitchenbot-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
prompt = "<|bos|>Simple Tomato Sauce\nIngredients: tomatoes, garlic, olive oil\nSteps:"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=False))
For Q&A chat, load this base plus the LoRA adapter — see bychwa/kitchenbot-chat.
Reproduce
Full scripts (uv setup, corpus → tokenizer → pretrain → SFT → CLI):
https://github.com/bychwa/kitchenbot
export HF_USER=bychwa
export WANDB_PROJECT=kitchenbot
python scripts/03_pretrain_base.py
License
Apache-2.0 for the model code/weights packaging in this card’s training setup. Respect the license/terms of idoyaaran/mise-recipes for the underlying text.