Muse-525M

A from-scratch roleplay and companion language model by BytesTalk. Not a fine-tune of anything — every weight was trained from random initialization, starting from a tokenizer built for this model and a corpus assembled for it.

Muse is the first model in a new family, separate from the PersonaMini-1 line. It is roughly 3× the size of the largest PersonaMini, has 2× the context, and speaks four languages instead of one.

18+. This model writes adult fiction. It refuses sexual content involving minors. It is a 525M model: it has limited world knowledge and states wrong facts confidently. It is for character roleplay and conversation, not for information.

Muse vs the PersonaMini-1 family

The mode token

Muse was trained with the content register as the first token of the sequence. This is not a system-prompt convention the model may or may not respect — it is a token it saw on every training example, and it changes what comes out.

Sending <|sfw|> and then asking for explicit content works against the model rather than with it. If the output feels tamer than expected, check which token the prompt starts with.

Prompt format

The exact layout the model was fine-tuned on. Deviating from it degrades output noticeably, because the special tokens — not text markers like ### USER: — are what the model learned:

<|sfw|> <|system|> {character card} <|user|> {message} <|assistant|>

The trailing <|assistant|> is the cue to answer. Generation stops at <|endoftext|> (id 0).

The character card

Cards are short and factual. What works best, measured on this model: a paragraph of who the character is, then a Canon: block of flat facts, then a scenario line — around 200–300 tokens.

Character: Mira (she/her)
A deadpan roommate with dry humour who secretly cares.
Canon:
- Works nights at a bookshop on Vine Street.
- Shares the flat with two others: Tomas (chef), Rey (student).
Scenario: the kitchen, late evening.

Longer, more novelistic cards measure worse than short factual ones on the same questions, which is the opposite of what most people expect. Keep them tight.

Training

Pretraining — 19.2B tokens, from random initialization

The corpus was assembled for this model rather than taken off the shelf. The largest components:

Final base validation loss 2.14 (perplexity 8.51).

Supervised fine-tuning — a few rounds, continuation from the previous each time

Each round was one epoch at a low learning rate, loss masked to assistant tokens only, packed to 2048. Rounds were compared against each other on a fixed probe suite and by reading real conversations, because the two disagree more often than is comfortable — a round can score better and read worse.

This release is the round that came out best. Validation loss 1.9547.

What the weaker rounds were weaker at, and why they were not shipped:

• Reading facts off the card. Asked who plays drums when the card says so plainly, some rounds answered with a plausible name rather than the written one, or invented a bandmate. This is the hardest behaviour in the whole project and no round solved it completely.
• Trading one thing for another. A round that improved factual recall would come back flatter in character, or better in character and worse at remembering what the user had said ten turns ago.
• Register drift. Some rounds answered in-character questions in a plain assistant voice, which reads as broken even when the content is right.

Usage

The checkpoint is raw PyTorch — a state_dict plus its config — not a transformers model. The model definition (model_muse.py) and the tokenizer are in this repo.

import torch, json
from tokenizers import Tokenizer
from model_muse import Muse, MuseConfig

ck  = torch.load("muse_525m.pt", map_location="cpu")
cfg = MuseConfig(**ck["cfg"])
model = Muse(cfg)
model.load_state_dict(ck["model"])
model.eval().cuda()

tok = Tokenizer.from_file("tokenizer.json")

def build(card, history, mode="<|sfw|>"):
    sid = tok.token_to_id
    ids = [sid(mode), sid("<|system|>")] + tok.encode(card).ids
    for role, text in history:
        ids += [sid("<|%s|>" % role)] + tok.encode(text).ids
    return ids + [sid("<|assistant|>")]

ids = build("Character: Mira (she/her)\nA deadpan roommate.",
            [("user", "hey, long day?")])

Decoding. temperature 0.85, top_k 50, top_p 0.95, no repetition penalty and no n-gram blocking. no_repeat_ngram_size cuts songs off after two lines, because a chorus repeats by design.

Languages

Muse was pretrained on English, Japanese, Czech and Slovak, and fine-tuned with conversations in all four. They are not equally strong:

• English is the strongest by a wide margin, and most of the fine-tuning data is English.
• Japanese holds its script and register well; some replies leak an English sentence.
• Slovak is solid, with occasional dropped diacritics.
• Czech is the weakest. Muse is feminine, and in Czech she frequently refers to herself with masculine verb forms (byl instead of byla). Diacritics are dropped more often than in Slovak.

If you need reliable Czech, this is not the model.

Limitations

• Facts. 525M parameters. It states wrong things confidently and cannot be relied on for information of any kind.
• Card reading. It does not reliably find facts written on its own character card. Asked who someone is, it will sometimes produce a name that fits the scene rather than the one written down. This is the model's biggest weakness for roleplay use, and it is a capacity limit as much as a data one.
• Memory. 2048-token context. Anything older falls out of the window unless the front-end re-injects it.
• Consistency. Names and details drift in long scenes.
• Speed. This implementation has no KV cache and recomputes the sequence each step. Generation is correct but not fast.
• Guardrails. Refusal of sexual content involving minors was trained in, not bolted on. There is no classifier and no filter. It has not been adversarially tested, and a creative jailbreak should be assumed to work.

Trained from scratch by BytesTalk.