Kiyo-65M is a decoder-only language model pretrained from scratch on 80B tokens drawn from FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train.
It is the smallest member of the Kiyo family and shares the design of Kiyo-135M: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary. Kiyo-65M scales that recipe down to 15 layers and a 512-dimensional residual stream, roughly halving the parameter count while keeping the same data mixture and training code. It is trained independently from a random initialization, not distilled or pruned from Kiyo-135M.
Model Details
| Field | Value |
|---|---|
| Parameters | 64,994,816 |
| Architecture | Llama-style decoder (SmolLM2 architecture) |
| Layers | 15 |
| Hidden size | 512 |
| Intermediate size | 1,216 |
| Attention heads | 8 |
| KV heads | 4 |
| Head dimension | 64 |
| Attention type | Grouped query attention |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Positional encoding | RoPE (theta 10,000) |
| Vocabulary size | 49,152 |
| Context length | 2,048 |
| Embeddings | Tied input/output embeddings |
| Training tokens | 79,999,533,056 |
| Optimizer steps | 152,587 |
| Tokens per optimizer step | 524,288 |
| Peak learning rate | 2.0e-3 |
| LR schedule | Warmup-stable-decay (1/sqrt) |
| Weight format | safetensors |
Training Data
| Source | Domain | Mixture share |
|---|---|---|
| FineWeb-Edu | General web text, education-filtered | 45.0% |
| DCLM-Baseline | General web text, high-quality filtered | 30.0% |
| Stack-v3-train | Source code | 17.5% |
| FineMath | Mathematical reasoning | 7.5% |
Benchmarks
Self-reported result from the official BananaMind Base Bench 1.1 script, measured on CUDA in float32.
| Model | Params | Overall Elo |
|---|---|---|
| Kiyo-65M | 65.0M | 1,067 |
| Rose-Mini | 49.4M | 1,037 |
| BananaMind-2-Medium | 49.6M | 1,034 |
| Supra-50M-Reasoning | 51.8M | 985 |
Figures for Rose-Mini, BananaMind-2-Medium, and Supra-50M-Reasoning are taken from the official BananaMindBench Leaderboard, all against the same BananaMind Base Bench 1.1 suite. Note that all three comparison models are 20–30% smaller than Kiyo-65M, so this is not a parameter-matched comparison.
Detailed Kiyo-65M result
| Category | Accuracy | z vs. chance | Elo | Significant |
|---|---|---|---|---|
| Language completion | 98.0% | +11.92 | 1,468 | * |
| World knowledge | 74.0% | +8.00 | 1,099 | * |
| Commonsense | 68.0% | +7.02 | 1,062 | * |
| Code completion | 62.0% | +6.04 | 1,193 | * |
| Context tracking | 48.0% | +3.76 | 949 | * |
| Logical reasoning | 42.0% | +2.78 | 1,007 | * |
| Quantitative | 34.0% | +1.47 | 918 |
* = passes 1.96σ vs. chance; n=50 per category
By difficulty
| Difficulty | Accuracy |
|---|---|
| Easy | 70.9% |
| Medium | 59.8% |
| Hard | 51.7% |
Summary
| Metric | Value |
|---|---|
| Parameters | 64,994,816 |
| Overall Elo | 1,067 |
| Chance floor | 805 |
| Above chance floor | +263 |
| Raw accuracy | 60.9% |
| 95% CI on accuracy | 55.7% – 66.0% |
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.
Usage
pip install -U transformers safetensors torch
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "DedeProGames/Kiyo-65M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()
prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
output = model.generate(
input_ids=input_ids,
max_new_tokens=64,
do_sample=False,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
This is a base model, not instruction-tuned — it continues text rather than following instructions. At 65M parameters and 80B training tokens it still produces fluent, grammatical continuations and remains strong on language completion, but accuracy falls off sharply on quantitative reasoning, multi-step logic, and context tracking, where it sits close to chance on the harder items. The 2,048-token context window also limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
License
Apache 2.0