KeyLM-75M-Instruct

KeyLM-75M-Instruct is a 75M parameter instruction-tuned language model trained from scratch on approximately 18 billion tokens. That training budget is a small fraction of what comparable small models use (SmolLM-135M was trained on roughly 600B tokens, SmolLM2-135M on roughly 2T). Despite this, it is competitive on instruction following, outperforming SmolLM-135M-Instruct on IFEval while using about half the parameters and a fraction of the data.

Table of Contents

1. Model Summary
2. How to Use
3. Evaluation
4. Training
5. Limitations
6. License
7. Citation

Model Summary

KeyLM is a compact decoder-only transformer built on the standard small-model recipe used by Llama and Qwen3: grouped-query attention, rotary position embeddings (RoPE), SwiGLU feed-forward layers, and per-head QK-RMSNorm. It is designed for lightweight, low-latency English chat and instruction following.

GGUF builds for llama.cpp, LM Studio, and Ollama are available at KeyLM-75M-Instruct-GGUF.

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Eclipse-Senpai/KeyLM-75M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.bfloat16
)

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)
outputs = model.generate(
    inputs, max_new_tokens=128, do_sample=True,
    temperature=0.7, top_p=0.9, repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Evaluation

Instruction following (IFEval)

This is where KeyLM is competitive. All rows are evaluated with lm_eval (ifeval, 541 prompts, greedy decoding).

KeyLM beats the original SmolLM-135M-Instruct at roughly half the size and a fraction of the training data. SmolLM2-135M-Instruct, a far more heavily trained model, remains ahead.

Base vs Instruct

The base and instruction-tuned checkpoints across all benchmarks. Commonsense and knowledge tasks are zero-shot via lm_eval (accuracy; ARC and HellaSwag length-normalized); IFEval is the 4-metric average.

Instruction tuning leaves knowledge and reasoning roughly unchanged; its real effect is the instruction-following ability IFEval captures. Both versions sit modestly above random on basic commonsense and at chance on MMLU.

Training

Pretraining

KeyLM was pretrained from random initialization on approximately 18B tokens, drawn from a weighted mixture of public datasets and streamed through a deterministic curriculum.

Post-training

Instruction tuning used smol-smoltalk, ultrachat_200k, and several smoltalk2 splits (magpie, persona instruction-following, science, OpenHermes, system chats, summarization), with assistant-only loss masking, plus a set of custom synthetic instruction-following examples.

Limitations

• Minimal world knowledge. Not suitable for factual question answering, reasoning, math, or code.
• English only.
• No dedicated safety alignment was performed. Apply your own filtering before any user-facing use.

License

Apache 2.0. The weights are trained from scratch and free to use, modify, and redistribute.

Citation

@misc{keylm75m2026,
  title  = {KeyLM-75M: a from-scratch small language model},
  author = {Eclipse-Senpai},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Eclipse-Senpai/KeyLM-75M-Instruct}}
}