lilbase
a 297m param llama-style base model trained from scratch on a kaggle tpu v5e-8.
its a base model, so it continues text instead of answering questions. give it the start of a sentence, not a question. grammar is solid, facts are confidently made up. for the chat version see lilchat.
run it
ollama run navthings/lilbase "The water cycle begins when"
or try it in your browser: https://navthings.github.io/playground/
files
| file | size | notes |
|---|---|---|
model.safetensors + config + tokenizer | 1.19gb | transformers LlamaForCausalLM |
lilbase-q8_0.gguf | 379mb | same perplexity as f16 |
lilbase-q4_k_m.gguf | 274mb | ~0.8% higher perplexity, smallest |
the model
gqa, rope, rmsnorm, swiglu. 24 layers, d=1024, 16 query heads, 4 kv heads, 1024 context. llama tokenizer, 32k vocab. tied embeddings.
trained on 6.1b tokens of fineweb-edu (sample-10BT), roughly chinchilla-optimal for this size. 11,043 steps of 524k tokens, data parallel over 8 tpu chips. held-out loss 2.608.
vs gpt-2 small
| benchmark | lilbase | gpt-2 124m |
|---|---|---|
| hellaswag (acc_norm) | 41.0% | 31.1% |
| arc-easy (acc_norm) | 50.5% | 39.5% |
| lambada | 28.7% | 32.6% |