Kiyo-230M-Preview is an intermediate training checkpoint of Kiyo-230M, a decoder-only language model being pretrained from scratch on FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train. It is released as a preview: the training run is still in progress and this checkpoint has seen roughly one fifth of its planned token budget. See Preview Status below before using or comparing it.
The architecture scales the Kiyo-135M design up to ~230M parameters: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary. Compared to Kiyo-135M, it goes deeper and wider — 32 layers at hidden size 768, against 30 layers at 576 — while keeping the same tokenizer, the same data mixture, and the same training code.
Preview Status
This repository holds the weights at optimizer step 81,500 of a run targeting 200B tokens. Three things follow from that, and all three matter when reading the benchmark numbers:
| Field | Value |
|---|---|
| Tokens seen by this checkpoint | 42,729,472,000 (42.7B) |
| Target token budget | 200,000,000,000 (200B) |
| Progress | 21.4% |
| Optimizer step | 81,500 |
| Planned total steps | ~381,470 |
| Learning rate at this checkpoint | 1.2e-3 (peak, stable phase) |
| LR decay begins at step | ~343,320 (final 10% of the run) |
The learning rate has not decayed yet. The run uses a warmup-stable-decay schedule whose decay phase covers only the final 10% of steps. This checkpoint sits deep in the stable phase, still at the peak LR of 1.2e-3. Models evaluated mid-plateau consistently score below what the same weights reach after annealing, so the numbers below understate the model rather than describe it.
It has seen fewer tokens than Kiyo-135M. Kiyo-135M was trained on the full 200B tokens. This preview has seen 42.7B. A parameter-count comparison between the two is not a fair comparison of the architectures — it is a comparison of a finished run against a fifth of one.
Weights are not final and will change. Do not treat this checkpoint as the Kiyo-230M release. It exists so the run can be inspected mid-flight; the final model will be published separately.
Model Details
| Field | Value |
|---|---|
| Parameters | 229,688,064 |
| Architecture | Llama-style decoder (SmolLM2 architecture family) |
| Layers | 32 |
| Hidden size | 768 |
| Intermediate size | 1,920 |
| Attention heads | 12 |
| KV heads | 4 |
| Head dimension | 64 |
| Attention type | Grouped query attention |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Positional encoding | RoPE (theta 10,000) |
| Vocabulary size | 49,152 |
| Context length | 2,048 |
| Embeddings | Tied input/output embeddings |
| Training tokens (this checkpoint) | 42,729,472,000 |
| Tokens per optimizer step | 524,288 |
| Peak learning rate | 1.2e-3 |
| LR schedule | Warmup-stable-decay (1/sqrt), decay not yet reached |
| Weight format | safetensors |
Training Data
| Source | Domain | Mixture share |
|---|---|---|
| FineWeb-Edu | General web text, education-filtered | 45.0% |
| DCLM-Baseline | General web text, high-quality filtered | 30.0% |
| Stack-v3-train | Source code | 17.5% |
| FineMath | Mathematical reasoning | 7.5% |
The Stack-v3 portion is sampled with per-language keep probabilities across 32 curated programming languages, giving each language an equal token quota rather than inheriting the corpus's natural skew.
Benchmarks
Self-reported result from the official BananaMind Base Bench 1.1 script, measured on CUDA in float32.
| Model | Params | Training tokens | Overall Elo |
|---|---|---|---|
| Kiyo-135M | 134.5M | 200B | 1,126 |
| BananaMind-2-Pro | 139.0M | 100B | 1,124 |
| Rose-Pro | 151.3M | — | 1,105 |
| Kiyo-230M-Preview | 229.7M | 42.7B | 1,086 |
| GPT-2 | 124M | 10B | 990 |
Figures for BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite. This preview sits below Kiyo-135M despite having 1.7× the parameters, which is the expected result at 21% of the token budget with the learning rate still at peak — see Preview Status.
Detailed Kiyo-230M-Preview result
| Category | Accuracy | z vs. chance | Elo | Significant |
|---|---|---|---|---|
| Language completion | 98.0% | +11.92 | 1,468 | * |
| Code completion | 74.0% | +8.00 | 1,286 | * |
| World knowledge | 72.0% | +7.68 | 1,079 | * |
| Commonsense | 70.0% | +7.35 | 1,077 | * |
| Context tracking | 44.0% | +3.10 | 931 | * |
| Logical reasoning | 44.0% | +3.10 | 1,017 | * |
| Quantitative | 38.0% | +2.12 | 958 | * |
* = passes 1.96σ vs. chance; n=50 per category
Every category clears the significance threshold, including Quantitative — the one category where Kiyo-135M does not.
By difficulty
| Difficulty | Accuracy |
|---|---|
| Easy | 73.5% |
| Medium | 60.7% |
| Hard | 54.3% |
Summary
| Metric | Value |
|---|---|
| Parameters | 229,688,064 |
| Overall Elo | 1,086 |
| Chance floor | 805 |
| Above chance floor | +282 |
| Raw accuracy | 62.9% |
| 95% CI on accuracy | 57.8% – 67.9% |
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. With 350 items, the 95% interval spans roughly ±5 accuracy points, so small Elo gaps between models should not be read as decisive.
Usage
pip install -U transformers safetensors torch
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "DedeProGames/Kiyo-230M-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()
prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
output = model.generate(
input_ids=input_ids,
max_new_tokens=64,
do_sample=False,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
This is an unfinished base model, not instruction-tuned — it continues text rather than following instructions, and its weights are an intermediate checkpoint rather than a release. Beyond the preview caveats above, it shares the limits of its scale: fluent and grammatical on language completion, competent on code, but weak on multi-step logic and context tracking, where it stays near chance on the harder items. The 2,048-token context window limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.
License
Apache 2.0