Moonfrost-777M-Instruct-v1

Code · Site · Training runs

A supervised fine-tune of Moonfrost-777M, not a separate model. It starts from that base and trains for 0.42 epochs on instruction data. Architecture, tokenizer and parameter count are identical; only the weights differ.

Two tunes of that base are published and both are usable; they differ in purpose rather than in kind. This one carries no identity data, so it claims no persona and holds no opinion about who built it, which makes it the neutral starting point for tuning your own behaviour on. Instruct-v2 is the one tuned for conversation: four times the exposure, plus the persona set.

Read on before building from it. Half an epoch is not much instruction tuning and it shows, in two ways. It drifts between speakers: it had not seen enough complete conversations to learn reliably where an assistant turn ends, so it will sometimes write your next message for you, which is why both <|endoftext|> and <|user|> have to be treated as stop tokens. And it has no identity, because the persona data was only added in v2. Asked who made it, this checkpoint answers that it is a retired professor of English named Jack Harris who has taught in the Midwest for about ten years, and then asks the question back. v2 answers in one line and stops. Asked for hello world in Java, this one produces a class containing two main methods, which will not compile, alongside an explanation about abstract base classes that has nothing to do with the code; v2 produces the correct four lines and stops cleanly.

Neither failure is a defect of the architecture or of the pretrained base. Both are what an undertrained supervised fine-tune looks like.

Architecture is documented in full on Moonfrost-777M.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M-Instruct-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

prompt = "<|system|>You are Moonfrost, a helpful assistant.<|user|>Why is the sky blue?<|assistant|>"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True,
                        temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Conversations are formatted <|system|>…<|user|>…<|assistant|>…<|endoftext|>, and stopping on <|user|> as well as <|endoftext|> matters more here than it does for v2.

Benchmarks

All three Moonfrost checkpoints were scored, so the two tunes can be read against the base they came from and against each other. Five-shot, 250 examples per benchmark, likelihood scored, with the reference models run through that same harness on those same examples rather than quoted from their cards.

The scores sit where they do because six billion tokens for 777 million parameters is about eight tokens per parameter, against a compute-optimal ratio of roughly twenty and against reference models that read two to eighteen trillion.

Intended use

Use these weights as an instruction-following starting point that holds no opinion about who it is. Nothing in the fine-tune mixture describes an identity, so a persona tuned on top of it has nothing to argue with. That is the one thing this checkpoint does better than v2.

Read the section below first. Half an epoch of instruction tuning leaves speaker roles unstable, which is a defect if you deploy it and mostly irrelevant if you tune over it.

Do not put it in front of users or anywhere an answer is acted on. It has no safety tuning and no content filtering, and it drifts between speakers often enough to produce dialogue nobody wrote.

Limits

Everything true of v2, and more of it. It fabricates facts confidently, performs no arithmetic or multi-step reasoning, handles English only, sees at most 1,024 tokens, and has no safety tuning, no RLHF and no content filtering of any kind. On top of that it drifts between speakers and has no self-description.

Treat it as something to tune from rather than something to deploy. For a conversation, take Instruct-v2.

Citation

@misc{kumar2026moonfrost,
  title  = {Moonfrost: A 777M Mixture-of-Experts Language Model
            Trained From Scratch},
  author = {Kumar, Ashish},
  year   = {2026},
  url    = {https://github.com/whoashish115/moonfrost-ai}
}

Apache 2.0.