Lattice Mini
A 42M-parameter GPT-2-style decoder-only language model trained from scratch by Lattice. No pretrained weights were used.
Specs
| Params | 42M |
| Architecture | GPT-2 (12 layers, 512 embd, 8 heads) |
| Context | 512 tokens |
| Vocab | 8,000 (BPE, byte-level) |
| Pretraining | WikiText-2, 10k iters |
| Chat fine-tune | Alpaca-style instruct data (5k examples, 1.2k iters) |
| Format | ### Instruction:\n...\n### Response:\n |
Honest expectations
This is a hobby-scale model. It was pretrained on WikiText-2 (a small corpus) and lightly fine-tuned on 5,000 Alpaca examples. It produces repetitive, low-quality text and is not usable for anything serious — it exists to document the training journey and to let people run a real from-scratch model.
Try it in the HF Space or read the write-up on the Lattice blog.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("lattice-research/lattice-mini")
model = AutoModelForCausalLM.from_pretrained("lattice-research/lattice-mini")
prompt = "### Instruction:\nWrite a haiku about trains.\n### Response:\n"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, temperature=0.4)
print(tok.decode(out[0], skip_special_tokens=True))
Weights
model.safetensors— 42M params, fp32- Trained with a custom nanoGPT-style implementation; converted to a
standard
GPT2LMHeadModel(bias-free) so it loads with plain transformers. - Uses a custom 8k BPE tokenizer (byte-level,
<|bos|> <|eos|> <|unk|> <|pad|>).
The Lattice lineup
- Mini (this) — 42M, from scratch
- Spark — 1.5B, Qwen fine-tune, the flagship
- Quark — 1.5B, from scratch, training in progress
More at lattice-research on Hugging Face and lattice-research.vercel.app.