TinyBalls-v1-Base
The base (non-instruct) model: 113.3M parameters, trained from scratch on
2.01B tokens of English text. For the chat and tool-use version
see igidn/TinyBalls-110M-V1.
Architecture
| params | 113.3M (151.0M counting the tied head twice) |
| layers | 12 |
| hidden size | 768 |
| attention | 12 query heads / 4 KV heads (GQA), head dim 64 |
| FFN | SwiGLU, intermediate 2048, pre-activations clamped at 10.0 |
| vocab | 49,154 (SmolLM2 byte-level BPE, plus ChatML and tool specials reserved but untrained) |
| context | 2,048 (the length it was trained at) |
| tied embeddings | yes |
| RoPE theta | 10,000 |
Weights are float32 and load as a standard transformers
LlamaForCausalLM.
Usage
This is a completion model. There is no chat template, and the ChatML tokens in
its vocabulary are untrained, do not prompt it with <|im_start|>.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("igidn/TinyBalls-v1-Base")
model = AutoModelForCausalLM.from_pretrained(
"igidn/TinyBalls-v1-Base", torch_dtype="float32")
ids = tok("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=48, do_sample=True,
temperature=0.8, top_p=0.95)
print(tok.decode(out[0][ids.shape[1]:]))
Documents were packed with <|endoftext|> (id 0) as the separator, which is
also the EOS and pad token here. The generation_config.json defaults
(temperature=0.8, top_p=0.95, sampling on) are what you want: greedy
decoding collapses into repetition on this checkpoint, under greedy sampling
the 4-gram repeat rate is 0.94 and distinct-token rate 0.04, versus 0.05–0.09
and 0.38–0.43 under top-p sampling.
Training
Two phases on a 2.01B-token English-only mix (1.5B broad, then a 0.5B quality anneal over the final 25% of steps), one LR schedule spanning both:
| domain | tokens |
|---|---|
| web (FineWeb-Edu) | 670M |
| books (Gutenberg, pre-1929, DOAB, LoC) | 150M |
| Wikipedia (FineWiki) | 130M |
| code (UltraData-Code, codeparrot) | 120M |
| math (FineMath, OpenWebMath, AutoMathText) | 110M |
| synthetic prose (Cosmopedia) | 80M |
| papers (peS2o, arXiv, Proof-pile-2) | 80M |
| QA and forums (Stack Exchange, Ubuntu IRC) | 80M |
| micro-domains (patents, legal, regulations, ...) | 80M |
Files
| file | |
|---|---|
model.safetensors | fp32 LlamaForCausalLM weights |
config.json | architecture, rope_theta 10000, 2048 context |
tokenizer.json | the exact tokenizer these weights were trained with |
tokenizer_config.json | specials; no chat template |
generation_config.json | sampling defaults, EOS = `< |
Modelfile | minimal Ollama serving file |
License
MIT.
-igidn