Kiyo-135M-0960 is an updated checkpoint of Kiyo-135M, continuing pretraining from 200B to 1,000,000,000,000 (1T) tokens on the same data mixture: FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train.
The model follows the SmolLM2-135M architecture: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings. SmolLM2 was chosen as a base architecture because it is specifically tuned for small-scale pretraining efficiency — narrow hidden size with a comparatively deep layer stack, and a large 49k-token vocabulary that keeps sequence lengths short for a model this size. Kiyo-135M-0960 uses the same architecture and initialization lineage as Kiyo-135M, extended with additional pretraining on the same data mixture.
Model Details
| Field | Value |
|---|---|
| Parameters | 134,515,008 |
| Architecture | Llama-style decoder (SmolLM2 architecture) |
| Layers | 30 |
| Hidden size | 576 |
| Intermediate size | 1,536 |
| Attention heads | 9 |
| KV heads | 3 |
| Attention type | Grouped query attention |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Positional encoding | RoPE (theta 100,000) |
| Vocabulary size | 49,152 |
| Context length | 8,192 |
| Embeddings | Tied input/output embeddings |
| Training tokens | 1,000,000,000,000 |
| Weight format | safetensors |
Training Data
| Source | Domain |
|---|---|
| FineWeb-Edu | General web text, education-filtered |
| DCLM-Baseline | General web text, high-quality filtered |
| FineMath | Mathematical reasoning |
| Stack-v3-train | Source code |
Benchmarks
Self-reported results from the official BananaMind Base Bench 1.1 script, all measured with the same runner, dtype (bfloat16) and GPU.
| Model | Params | Overall Elo |
|---|---|---|
| Kiyo-135M-0960 | 134.5M | 1,139 |
| Kiyo-135M | 134.5M | 1,126 |
| BananaMind-2-Pro | 139.0M | 1,124 |
| Rose-Pro | 151.3M | 1,105 |
| GPT-2 | 124M | 990 |
Figures for Kiyo-135M, BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite.
Detailed Kiyo-135M-0960 result
| Category | Accuracy | z vs. chance | Elo | Significant |
|---|---|---|---|---|
| Language completion | 98.0% | +11.92 | 1,468 | * |
| Code completion | 88.0% | +10.29 | 1,453 | * |
| World knowledge | 82.0% | +9.31 | 1,177 | * |
| Commonsense | 80.0% | +8.98 | 1,178 | * |
| Logical reasoning | 50.0% | +4.08 | 1,061 | * |
| Context tracking | 52.0% | +4.41 | 1,002 | * |
| Quantitative | 28.0% | +0.49 | 889 |
* = passes 1.96σ vs. chance; n=50 per category
By difficulty
| Difficulty | Accuracy |
|---|---|
| Easy | 76.1% |
| Medium | 65.0% |
| Hard | 63.8% |
Summary
| Metric | Value |
|---|---|
| Parameters | 134,515,008 |
| Overall Elo | 1,139 |
| Chance floor | 805 |
| Above chance floor | +334 |
| Raw accuracy | 68.3% |
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.
Usage
pip install -U transformers safetensors torch
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "DedeProGames/Kiyo-135M-0960"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()
prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
output = model.generate(
input_ids=input_ids,
max_new_tokens=64,
do_sample=False,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
This is a base model, not instruction-tuned — it continues text rather than following instructions. At 135M parameters it produces fluent, well-structured text and is strong on language completion and code, but accuracy drops on quantitative and context-tracking tasks. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
License
Apache 2.0