ali-llm v0.1

ali-llm is a 2.05B-parameter decoder-only language model trained from scratch on English text written for children, in the order a child would meet it: pre-kindergarten speech and stories first, then kindergarten, then grades 1 to 5 in turn. Reading, arithmetic, elementary science and everyday money skills (counting coins, earning, spending, saving) are all part of the curriculum, and the difficulty of what the model sees only ever goes up during training.

The model saw 6.3 billion tokens (2.3 billion unique) and is released as a base model, with no instruction tuning and no safety tuning. It is a research artifact for studying curriculum-ordered pretraining and language acquisition, and a strong small model for children's-level text, simple grammar and grade-school arithmetic.

How to use

The architecture has a per-head output gate that no built-in transformers class provides, so the modeling code ships with the weights and trust_remote_code=True is required for both the model and the tokenizer (the tokenizer itself is a plain tokenizers BPE file; the flag is only needed because AutoTokenizer reads the model config). Tested with transformers 5.17 and torch 2.13.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "aliwister/ali-llm-v0.1"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()

prompt = "Once upon a time, there was a little dog named Max."
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

The tokenizer adds no special tokens. Documents in training were separated by <|endoftext|> (id 0), which also serves as the end-of-sequence token; <|pad|> is id 1. Prompts work best when they look like the training text: a story opening, a lesson, a question in the form Question: ...\nAnswer:, or an arithmetic drill such as 7 x 8 = .

Model details

The design is a Llama-style decoder plus the two attention additions (QK-norm and per-head output gating), both of which are known to stabilise training at high learning rates. Gates were initialised at zero, so every head started half open.

Tokenizer

Byte-level BPE with 49,152 tokens, trained with the tokenizers library. Every digit is its own token and pre-tokenization follows the GPT-4 style word-splitting pattern. The training sample mixed the children's text below with a broader sample of English (Wikipedia, textbooks, educational web pages, corporate filings and forum discussion) so that the vocabulary is not limited to children's language. 16 special tokens: <|endoftext|>, <|pad|> and 14 reserved.

Training data

All training text was assigned a level on a 7-step scale, pre-K, K, grades 1 to 5, and the model read the levels in order. Every source covers a range of levels; within a source, documents were sorted by Flesch-Kincaid reading grade and split evenly across the source's range. Math documents were additionally kept at or above the lowest grade that teaches the ideas they use, ScienceQA used its own grade labels, and each arithmetic drill used the grade of the skill it practises.

Unique training tokens per source and level, in millions (our tokenizer):

96.7% of the unique text was written by a language model (TinyStories, Cosmopedia, Orca math, the drills). Real text (CHILDES, Gutenberg, Simple Wikipedia, ScienceQA) is about 77 million tokens.

Filtering

A document was dropped if it

• uses an idea taught in grade 6 or later (for example quadratic equations, atoms or interest rates), detected with word lists for math, science and everyday money topics;
• has a reading score far above its source's range;
• is shorter than 20 words (except ScienceQA questions and drills);
• is an exact duplicate of another document;
• shares any run of 13 tokens with a question from the evaluation sets below.

The largest cut was to Cosmopedia v1 stories for young children: every one that passed the other filters was a duplicate of a Cosmopedia v2 story, so none were kept.

Arithmetic drills

A generator writes practice text for 71 skills, most taken from the Common Core math standards for kindergarten to grade 5 (counting, addition and subtraction within 10, 20 and 100, place value, multiplication tables, division, fractions, measurement, time and money) plus pre-K counting and four grade-5 money skills (earning, spending, saving, budgets). Each document explains a skill, works an example and then asks questions with answers. Every generator was checked against an independent solver on 12,000 random draws per skill, and every equation in 1,050 sampled documents was recomputed and found correct.

Ordering and repetition

• Levels were trained strictly in order, pre-K first, with documents shuffled within a level.
• Within each level after pre-K, 15% of rows were review text drawn from the levels already finished and spread evenly through the level, so earlier material was not forgotten. Review never included anything above the current level.
• Documents were packed into rows of 2,048 tokens separated by <|endoftext|>.
• To reach the token budget, most sources were read 2.81 times. TinyStories was capped at 2 passes because its style is very uniform, and drills were read once.
• About 1.5 million tokens per level were held out and never trained on.

Training procedure

Loss on held-out text was measured every 500 steps for every level, and the model was tested on BLiMP, ScienceQA and arithmetic drills at the same points. A checkpoint was kept from just before the learning-rate decay ("pre-decay", step 11,520) as well as the final one; both are reported below.

Evaluation

All tests are zero-shot. The SmolLM2 models are included as reference points, run through exactly the same tests. They were trained on 2 to 11 trillion tokens of mostly adult web, code and math text, hundreds to thousands of times more than this model. SmolLM2-1.7B is the closest in size to ali-llm (1.7B vs 2.05B parameters) and saw about 1,700 times more data, so it is the most informative column: it shows what the same-sized model gives up, and gains, by reading only 6.3 billion tokens of children's text.

How each test was run:

• BLiMP: a pair counts as right when the grammatical sentence gets the higher total log-probability.
• ScienceQA: the model picks the choice whose text is most likely after the question (summed log-probability of the choice tokens). The grade 1 to 5 subset matches what the model was trained on; the higher grades were never seen.
• Arithmetic: greedy decoding, up to 12 new tokens; the text before the first newline must match the answer exactly. Problems from skills with a small problem space (for example single-digit multiplication) were likely seen in training and are reported separately below.
• ASDiv and GSM8K: prompt Question: <question>\nAnswer:, greedy decoding, up to 256 new tokens; the last number in the answer must equal the reference.
• Bits per byte: on about 1.5 million held-out tokens per level, so it is comparable across tokenizers.

Against the same-sized SmolLM2-1.7B, ali-llm is ahead on arithmetic and ASDiv, close on GSM8K and ScienceQA, and behind on grammar. By level, ali-llm is far more fluent on pre-K to grade 1 text and SmolLM2-1.7B is more fluent from grade 2 up.

Held-out text by level

Bits per byte, lower is better.

BLiMP by field

Arithmetic drills by level

Held-out problems, never seen in training:

Problems from small skill spaces, likely seen in training (1,671 problems): 46.4% final, 49.7% pre-decay, and 22.7%, 25.0% and 0.8% for SmolLM2-1.7B, 360M and 135M.

The pre-decay checkpoint scores higher on the arithmetic drills and slightly higher on BLiMP, while the final checkpoint is better at word problems, science questions and held-out text at every level below grade 5. Two things are going on. First, format: the final model often answers in a full sentence ("The number before 10 is 9.") that the exact-match scorer rejects; a scorer that accepts the last number of the first sentence narrows the gap at grades 1 to 3 to nothing. Second, a real loss at the top of the curriculum: under that lenient scoring the final model still gets 110 grade-5 and 70 grade-4 drill problems wrong that the pre-decay model got right (against 23 and 30 the other way), with wrong numbers rather than wrong format, mostly in order of operations, decimals and the money problems. The last 1,280 steps of the schedule fell entirely inside the grade-5 text, on the third pass over the same documents; training loss fell from 1.29 to 1.00 while grade-5 held-out loss rose from 1.50 to 1.60, so the model was memorising those documents rather than the skills. A schedule that ends on a mix of all levels, or fewer passes over the top level, should avoid this.

Limitations

• The model knows children's-level English only. It has not seen news, code, adult non-fiction, dialogue beyond children's stories or anything requiring knowledge taught after grade 5, and questions on such topics produce confident nonsense.
• 96.7% of the training text was generated by other language models, so the model inherits their style, and its output can be repetitive, especially with greedy decoding. Sampling with temperature around 0.8 and top-p 0.95 gives more natural text.
• It is a base model: it does not follow instructions or chat, and it has had no safety tuning. Its training data is children's material, so toxic output is unlikely but not impossible, and it can still produce incorrect facts and arithmetic.
• Level assignments inside each source rest on a readability formula, so text of quite different difficulty can share a level.
• Trained at a context of 2,048 tokens. Longer inputs up to 4,096 run but are untested.

Release contents

• model.safetensors: bfloat16 weights, 4.1 GB.
• config.json, configuration_ali.py, modeling_ali.py: the model definition for transformers.
• tokenizer.json, tokenizer_config.json: the tokenizer.
• generation_config.json: default generation settings.

Citation

@misc{ali-llm-v0.1,
  title  = {ali-llm v0.1: a 2B language model trained in grade order on children's text},
  author = {Al-Lawati, Ali},
  year   = {2026},
  url    = {https://huggingface.co/aliwister/ali-llm-v0.1}
}