KLM 1.1

KLM 1.1 is an 88M-parameter base language model, the continued training of KeyLM-75M. It scores 17.40 on the Open SLM Intelligence Index, up from KeyLM-75M's 10.44. That moves it from 38th to 9th among models under 100M parameters on the Open SLM leaderboard.

KeyLM is now KLM. This release keeps the KeyLM architecture and continues from its weights.

Why KLM 1.1 exists

KeyLM-75M was the first model we trained, and it was flawed in many ways. Most of its data was casual chat (Reddit, WildChat, LMSYS and similar), its tokenizer was small, and it stopped at about 18B tokens while still at a high learning rate. It is currently 38th on the leaderboard among models under 100M parameters.

The goal of KLM 1.1 was to find out how far that same base model could be pushed by more training alone, without changing its architecture. Over about 120B more tokens, we gave it a larger tokenizer, retrained it on a cleaner, education-, math- and reasoning-heavy mix, and ended with a proper learning-rate decay.

The gain shows up on every benchmark, most on ARC-Easy (+8.6 points) and ArithMark-3 (+5.7).

How it was trained

KLM 1.1 went through three stages after KeyLM-75M, each continuing from the last one's weights:

Int Index values in this table are from our own evaluation runs. Our harness scores KeyLM-75M at 10.47; the leaderboard lists 10.44.

Held-out loss fell at every evaluation of the long continuation, from 2.131 to 1.814 nats per token, averaged over held-out text from each of its 11 data sources. Most of the benchmark gain arrived in the final learning-rate decay.

Tokenizer expansion

The 24,576-token vocabulary keeps all of KeyLM-75M's tokens and adds 12,299 new byte-level BPE merges learned on the training mix. Each new token starts from the average of the old tokens it replaces. The larger vocabulary encodes the same text in 1.31–1.41× fewer tokens, with the biggest savings on code and math. This stage trained on UltraX Ultra-FineWeb web text (80%), UltraData-Math (15%) and Code-Reasoning (5%).

Long continuation

In the long continuation, the learning rate warmed up to 3e-5, held there for the first 80% of the run (75.9B tokens), then decayed linearly to 10% of peak over the final 20% (19.0B tokens). The decay used a mix heavier in math and textbooks.

All sources were filtered before training. The filters dropped documents the model had already trained on, exact repeats, empty documents, and any document sharing a 13-word sequence with an item from the five evaluation sets.

Evaluation

All scores are zero-shot. ARC, HellaSwag and PIQA use lm-eval 0.4.12 with length-normalized accuracy. ArithMark-3 uses the official ArithMark-3.0 script. Our harness reproduces the leaderboard's published KeyLM numbers to within ±0.17 points per task.

Architecture

KLM 1.1 keeps the KeyLM-75M architecture unchanged. Only the embedding and output layers grew with the vocabulary, from 75.3M to 88.1M parameters.

The weights load as a standard Transformers Qwen3ForCausalLM, since the network computes exactly what that class computes. No custom code is needed, and tools that support Qwen3 can run it.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MinimaLabs/KLM-1.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)

inputs = tokenizer("The three primary colors are", return_tensors="pt")
outputs = model.generate(
    **inputs, max_new_tokens=40, do_sample=True,
    temperature=0.7, top_p=0.9, repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• Base model only. KLM 1.1 completes text. It is not instruction-tuned and does not hold a conversation. The tokenizer has ChatML tokens (<|im_start|>, <|im_end|>), but the model never saw them in training.
• Small model. Its world knowledge is thin and its facts are often wrong. Do not rely on it for factual answers, math or code.
• English only, with a 2,048-token context window.
• No safety alignment. Add your own filtering before any user-facing use.

License

Apache-2.0. See LICENSE.

Citation

@misc{klm11_2026,
  title  = {KLM 1.1: continued training of KeyLM-75M},
  author = {Eclipse-Senpai},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/MinimaLabs/KLM-1.1}}
}