Moonfrost-777M

Code · Site · Training runs

Moonfrost is a 777M-parameter Mixture-of-Experts language model written and trained from nothing: its own byte-level tokenizer, its own attention and routing code, its own training loop. No base model was adapted and no weights were borrowed. Pretraining took about ten GPU-hours on a rented H100 and read roughly six billion tokens of FineWeb-Edu.

This repository holds the base model: next-token prediction and nothing else. It has no chat template, no instruction tuning and no alignment, so it continues text rather than answering questions. Prompt it with the opening of a passage, not an instruction.

Two supervised fine-tunes start from these weights and differ from them only in the weights, sharing this architecture, tokenizer and parameter count: Instruct-v2, the one to use, and Instruct-v1, kept for comparison. For conversation, take v2.

Usage

The architecture is not part of transformers, so the repository ships its own modelling code and needs trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

inputs = tokenizer("The water cycle begins when", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=True,
                        temperature=0.8, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

Two ideas from DeepSeek-V2 do the work, and both trade stored parameters for cheap ones.

Multi-head Latent Attention replaces the usual per-head key and value cache with a single shared latent. Ordinary attention at fourteen heads and head dimension 64 caches 1,792 numbers for every token; this projects the input down to one 320-number latent, caches that, and reconstructs keys and values when it needs them. Position is the complication, because rotary embeddings rotate a key by where it sits and a rotated key cannot be rebuilt from an unrotated latent. So position travels separately, on a 32-number rotary key shared across all heads, and attention runs over the two concatenated. The cache holds 352 numbers per token instead of 1,792, roughly five times smaller. At inference the up-projection matrices fold into the query and output projections once, which is exact because both are linear, and after that the model never reconstructs keys and values at all.

DeepSeekMoE makes the feed-forward layers sparse. Every layer above layer 0 holds 32 routed experts and one shared expert; a router scores the token against all 32, the top 3 run, the shared one always runs, so a token passes through four of thirty-three. Layer 0 is a plain SwiGLU feed-forward because routing from the very first layer destabilised early training and one dense layer costs almost nothing. A load-balancing auxiliary loss at weight 0.01 keeps the router from collapsing onto a few favourites.

The implementation detail that mattered most was mechanical rather than mathematical. The experts are stored as three stacked (32, hidden, ffn) tensors and dispatched by capacity in the GShard style, so all 32 run as three batched matrix multiplies. The first version looped over experts in Python, which forced a GPU synchronisation thirty-two times per layer per step and ran at 6,056 tokens per second. Same mathematics, same results, but the stacked version runs at 179,000.

Beyond those: RMSNorm with no bias terms anywhere, SwiGLU activations, RoPE at theta 10,000 computed in float32 and cast back because bfloat16 lost enough precision at long positions to matter, and tied input and output embeddings sharing one 32,768 x 896 matrix, which saves 29M parameters.

Seventy per cent of the model is experts that stay idle for any given token. That is the whole trade: the capacity of a 777M model at roughly the compute of a 161M one.

Training

Pretraining ran in two phases on two separate machines, reading disjoint shards so no document was seen twice. Phase 1 took shards 0-2 over 276 minutes for about 2.9B tokens; phase 2 took shards 3-7 over 340 minutes for about 3.1B and ended at validation loss 2.976.

Splitting one annealing schedule across two machines works because the learning rate is parameterised by elapsed fraction of training rather than by step. Phase 2 started at fraction 0.5227, exactly where phase 1 stopped, so the cosine curve continued rather than restarting. Only the weights crossed the boundary; the optimizer state stayed behind, which is why phase 2 re-warms for 150 steps.

The charts show what was logged, which is less than what was run. The first pretraining attempt died at step 1,730 when its client connection dropped, and only the first 105 minutes of phase 2 were retrieved from the volume, so those panels are partial by construction and labelled as such. The losses and step counts in the table above come from checkpoint metadata, which is complete. The full step-by-step history for all three runs is on Weights & Biases.

Benchmarks

Every number was measured on one machine with one harness, five-shot, 200 examples per benchmark, scored by which answer option the model finds most likely. The reference models were run through that same harness on those same examples rather than quoted from their cards, because prompt wording and length normalisation move these scores by several points. Qwen2.5-0.5B shows why that matters: its published MMLU is 47.5, and it scores 34.4 here.

Six billion tokens for 777 million parameters is about eight tokens per parameter against Chinchilla's compute-optimal twenty, and the models in that table read two to eighteen trillion at half the size. That ratio, not the architecture, explains almost everything the model gets wrong.

Almost every difference between the three Moonfrost columns is inside the noise. The one that is not is ARC-Easy, falling 54.8, 52.4, 44.4 across these weights and the two tunes, which is what chat formatting costs on a multiple-choice stem. These weights are the leftmost column, and nothing in the table is below chance. The noise floor, and how far these scores move with the five-shot prompt, are in the benchmarks document.

Intended use

Use these weights to continue text, to measure what six billion tokens buys at this size, or as the starting point for a fine-tune of your own. They are the parent of both published tunes and carry no chat template, so a prompt shaped like an instruction gets a continuation of the instruction rather than an answer.

Two things are worth building on top. Instruction tuning is the obvious one, and the two published tunes show what roughly half an epoch and roughly two epochs each produce; the scripts that ran them are in the repository. Domain adaptation is the other, since a model that has read only educational web text has room to move on any corpus you can supply.

Do not put it in front of users, in a product, or anywhere an answer is acted on. It has no safety tuning and no content filtering, and at this scale a confident wrong answer is the common case rather than the edge case. It is a research artefact.

Limits

It invents facts with complete confidence, most often on the topics a corpus of educational web text does not cover. It cannot do arithmetic or multi-step reasoning, handles English only, and sees at most 1,024 tokens at a time. There is no safety tuning, no RLHF and no content filtering at either the data or the output stage. This is a working demonstration of a complete training pipeline at small scale, not a deployable model.

References

The implementation was written from scratch against these papers rather than adapted from released code.

Citation

@misc{kumar2026moonfrost,
  title  = {Moonfrost: A 777M Mixture-of-Experts Language Model
            Trained From Scratch},
  author = {Kumar, Ashish},
  year   = {2026},
  url    = {https://github.com/whoashish115/moonfrost-ai}
}

Apache 2.0.