smallm-125m

smallm-125m is a bilingual LLM trained on approximately 3.5B tokens of which 60% were English and 40% German. The model was completely trained from scratch on 2 Kaggle T4 GPUs over the span of 28 hours.

Since all of the training was done on Kaggle, the model training was completely free. You can check the training code out yourself at: https://github.com/He-Tag/smallm

Training was two stages: 3.49B tokens of pretraining, then instruction tuning on 162M tokens of English and German conversations. The context window is 1024 tokens for prompt and answer together.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("He-Tag/smallm-125m", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("He-Tag/smallm-125m", trust_remote_code=True)

messages = [{"role": "user", "content": "What is the capital city of France?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                       return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Note that the model can and will hallucinate when asked for facts that are not common knowledge.

trust_remote_code=True is needed for both the model and the tokenizer: the architecture is not one of the built-in transformers models. For the same reason the inference widget on this page does not work — the Hub does not execute custom code.

The defaults in generation_config.json (temperature 0.3, top-p 0.9, repetition_penalty 1.15, no_repeat_ngram_size 4) come from the evaluation rather than from habit: at temperature 0.7 the model invents facts it gets right at 0.3, and without n-gram blocking it loops on the prompts beyond its reach.

Write German with proper umlauts. "Österreich" is answered correctly with Vienna; "Oesterreich" is read as "Osteria" and answered with Paris. The tokenizer was trained on real German text, where that transliteration barely occurs.

Architecture

12 layers, 12 heads, width 768, vocabulary 32,768 (byte-level BPE trained on the training mixture), 135.3M parameters including embeddings.

Follows modded-nanogpt: RMSNorm without a learnable gain, no biases, rotary embeddings, QK-norm, ReLU² MLP, value-residual learning, U-net skip connections between the layer halves, a per-block embedding shortcut, zero-initialised output projections and a tanh logit softcap. The model does not use bf16, instead it uses fp16 with a gradient scaler due to the T4 having no native bf16 support.

What it can do

Measured on 22 prompts, greedy, no repetition penalty:

• Short facts with a clean ending: "Die Hauptstadt von Österreich ist Wien." in eight tokens, then it stops. 17 of 22 answers end on their own.
• Explanations that are correct where the topic is common — photosynthesis comes back with chloroplasts, chlorophyll and oxygen as a byproduct.
• Formats: numbered lists, email skeletons, summaries.
• Multi-turn: a reference back to an earlier turn holds for two or three rounds.

On Apple silicon via MPS it runs at around 90 tokens/s, and is usable on CPU.

Measured on text typed by hand, which no corpus can contain: 0.817 bits/char for German, 0.988 for English.

Limitations

• Arithmetic fails. "3 apples plus 2" produces "3 × 2 = 6".
• Translation does not exist. Asked to translate a German sentence, it returns the German sentence.
• Counting instructions fail. "Count from 1 to 10" turns into an essay about the number 1.
• Facts collapse beyond the most common ones. Vienna, Berlin and Shakespeare are right; the largest ocean comes back as the Great Barrier Reef.
• It hallucinates confidently instead of declining to answer.
• Long answers drift. Holding an intent across several paragraphs is where it breaks: asked for a polite cancellation email, it writes an enthusiastic acceptance.
• Numbers in multi-turn answers are usually wrong.

These are limits of scale, not of data quality. The model saw 3.5B tokens; comparable models of this size see several hundred billion.

Not suitable for: unverified factual lookup, translation, arithmetic, or anything with legal, medical or financial consequences.

License and data provenance

Weights and code are released under the Apache License 2.0.

The model was trained on the datasets below. None of them is copyleft; all require attribution, which this section provides.

Pretraining

• HuggingFaceFW/fineweb-edu — ODC-BY 1.0
• HuggingFaceTB/smollm-corpus (cosmopedia-v2) — ODC-BY 1.0
• epfml/FineWeb2-HQ (deu_Latn) — ODC-BY 1.0

Fine-tuning

• HuggingFaceTB/smol-smoltalk — Apache-2.0
• HuggingFaceTB/smoltalk2 — no license stated on the dataset card
• mayflowergmbh/alpaca-gpt4_de — none stated; derived from alpaca-gpt4 (CC BY-NC 4.0)
• mayflowergmbh/oasst_de — none stated; derived from OASST1 (Apache-2.0)
• DiscoResearch/germanrag — CC BY 4.0

Two things for anyone building on this:

• The English fine-tuning data contains answers generated with Llama-3.1-405B-Instruct (Smol-Magpie-Ultra, part of SmolTalk).
• One German source derives from data under CC BY-NC 4.0. If you need a commercially clean lineage, repeat stage 1 without it.

Released for research and education, without any warranty.