Mume English 125M

Mume English 125M is a 134M-parameter English language model trained from scratch on 1.33B tokens of FineWeb. With Muon, rotary positions, QK-norm, an untied head and squared ReLU it reads FineWeb validation text 3.9% better than its plain-AdamW calibration run (v0.1.0) on the same data and steps, and beats the plain-AdamW GPT-2 124M reference at equal tokens by 3.7%.

From Muse Mesh (Hugging Face). Part of the Muse Mesh English and Math models (collections). Every training run, including the ones not released, is logged at saansar.com, the Sansar Lab.

Model

Why this run: The EN1 recipe screens (Muon, an architecture bundle, both, both plus warm-down and value embeddings; 10,000 steps each against a same-schedule AdamW control) picked Muon + the architecture bundle; this is that recipe on EN0's exact data, steps and schedule.

Evaluation

Metric: bits per UTF-8 byte (lower is better): the model's negative log-likelihood of a text divided by the text's UTF-8 byte count, so models with different tokenizers share a denominator. Each set's records are joined with </s> into one stream and scored teacher-forced in 1,024-token windows with stride 512 (every scored token after the first window has at least 512 tokens of context); </s> carries loss but no bytes. Scored with scripts/train/eval_bpb.py on a GPU in bf16 autocast. The sets are frozen and published as MuseMesh/mume-eval-suites.

With 512-token windows (stride 256) the same sets read fineweb_val 1.1146, fineweb_val_clean 1.1306, wikitext103_test 1.2040, enwik8_test 1.4031, text8_test 1.2700 (eval/bpb_ctx512.json).

Against GPT-2 124M at the same token budget

The pass mark was fixed before training: the plain-AdamW GPT-2 124M run of the modded-nanogpt speedrun (record 2024-06-06; GPT-2 tokenizer, 524,288 tokens per step), read off its validation curve at our budget of 1,332,314,112 tokens: 1.1755 bits per byte on the same FineWeb validation shard (val loss 3.6143 nats per GPT-2 token, converted with the measured 4.43604 bytes per GPT-2 token). Their number is over non-overlapping 1,024-token windows, so the like-for-like comparison is our run's own stream validation, scored the same way; the sliding-window fineweb_val above gives every token more context and reads lower.

As recorded in the experiment log (EN1-full): stream val 1.1322 vs EN0 1.1781 (-3.90%); frozen fineweb_val 1.1062 vs 1.15174 (-3.95%); every set -3 to -4%. Step for step against EN0 (same data, schedule and steps) the gap is -6.3% at step 2,000, narrowing to -3.9% at the end. It beats the plain-AdamW GPT-2 124M reference at equal tokens (1.1322 vs 1.1755, -3.7%) and matches it at twice the tokens (1.135 at 2.62B).

For context, the published GPT-2 117M zero-shot numbers (OpenAI 2019, trained on WebText, about 10x our tokens): enwik8 1.16 bits per byte, text8 1.17 bits per character, WikiText-103 perplexity 37.5 (about 0.997 bits per byte). enwik8 is raw Wikipedia XML, far from FineWeb's text, which is why both models read high there.

Contamination: FineWeb contains copies of some evaluation text. Measured GPT-2-paper style (lowercase word 8-grams of each set looked up in the whole training slice): fineweb_val 3.7% of 8-grams, 3,935 of 14,983 records with a copied passage (a run of 5+ consecutive hits), wikitext103_test 1.7% of 8-grams, 18 of 62 records with a copied passage (a run of 5+ consecutive hits). fineweb_val stays as the like-for-like set (llm.c and the speedrun train on FineWeb with the same copies); fineweb_val_clean drops every document with a copied passage and is the honest generalisation number. GPT-2 measured 0.7% for WikiText-103 test against WebText.

How the recipe was chosen (EN1 screens)

Each screen changed one thing on top of EN0 and ran 10,000 steps (491.5M tokens) with an annealed schedule; it is read against an AdamW control on the same schedule (stream validation, bits per byte, at step 10,000).

EN1-full (this version) is muon_arch at EN0's full length. The untied head adds 24.6M parameters, which is part of the gain; the parameter table above reports totals with and without it.

Checked before release

• modeling_mume.py against the training code (scripts/train/model.py), fp32 on CPU, 6 inputs (one evaluation text per set, cropped to 1,024 tokens, and 1,024 random ids): with the same bf16 weights the logits are identical (max |difference| 0); against the fp32 training weights the bf16 storage moves logits by at most 0.269.
• Bits per byte with eval_bpb.py's scoring on the first 20 records of each set (each cut to 20,000 characters): 1.18894 (training checkpoint, fp32) vs 1.18901 (this repo, bf16 weights) = +0.0058%.
• The whole wikitext103_test set through this repo's model (bf16 weights, fp32 on CPU): 1.16528, against 1.16525 in the table (training checkpoint, GPU bf16 autocast), +0.003%.
• Tokenizer: AutoTokenizer(..., trust_remote_code=True) gives the training ids on 62/62 sample texts and decodes them back exactly on 62/62; on all 58,338 texts of the tokenizer's own check the same holds (see MuseMesh/mume-tokenizer-32k).
• Left-padded batches give the same logits as single sequences (max |difference| 2.8e-05).

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "MuseMesh/mume-english-125m"
tok = AutoTokenizer.from_pretrained(repo, revision="v0.2.0", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, revision="v0.2.0", trust_remote_code=True)

inputs = tok("The history of the city begins", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=100)
print(tok.decode(out[0], skip_special_tokens=True))

Needs transformers (tested with 5.18.0), torch and sentencepiece. trust_remote_code=True runs two small files from this repository, modeling_mume.py (the network; the Sansar model code with the classes renamed) and tokenization_mume.py (SentencePiece, so the ids are exactly the training ids); read them before you run them. Without trust_remote_code the tokenizer falls back to tokenizer.json, which agrees on short texts but splits a few near-tie words differently in long documents. The weights are stored in bfloat16; pass dtype=torch.float32 for fp32, usually the faster choice on a CPU. The tokenizer adds no BOS/EOS: training documents were separated by </s> only.

What this checkpoint wrote for that prompt (CPU, torch.manual_seed(0), 60 new tokens):

The history of the city begins sometime around 1170. It is a great city for a number of reasons:
- It is well known for being the home of the first two thefts - and the historical and cultural center of the city. The 100th anniversary of the attack on the

and with greedy decoding:

The history of the city begins with the 18th century, when the city was founded by the kings of the kings of the kings of the kings of the kings of the kings of the kings of the kings of the kings of the

To score text instead of generating it, call the model with labels=input_ids; loss is the mean negative log-likelihood per token in nats.

Training data

FineWeb sample-10BT files 001-004: 4,180,465 web documents, 2.16B words, 12.9 GB. File 000 is not used: llm.c's validation shard (our fineweb_val) is cut from its head, and the slice builder proved the cut aligned (decoded validation record 0 = parquet row 0). Files 005 and 007 were in the tokenizer's sample only.

Preparation (scripts/train/prep_data.py): Unicode NFC; documents longer than 20,000 characters cut into parts at whitespace (4,257,824 records); tokenized with the shared 32k tokenizer; one </s> after every record. A 0.5% validation stream by hash of each record's text (21,358 records); the rest is training: 4,236,466 records, 3,253,318,552 tokens, 12.84 GB (3.95 bytes per token).

Every training document is listed (source file, row, id; no text) in MuseMesh/mume-eval-suites, config training_manifests, split english_pretrain, with the parts that went to validation, so the slice can be rebuilt from the public source.

Training

Limitations

• Base model. It continues text. It has not been instruction-tuned or aligned: it does not follow instructions, answer questions or chat, and it will not refuse anything.
• It makes things up. Fluent text with invented facts, names, numbers, citations. Do not use it as a source of facts.
• Small and short. 134M parameters and a 1,024-token context. This implementation has no key/value cache, so long generations are slow (each new token re-reads the context).
• Web English. FineWeb is filtered Common Crawl text: the model reproduces its topics, styles, biases and errors, including offensive content that passed the filters. English only; other languages appear only where FineWeb's English pages contain them.
• Contamination. Some evaluation text occurs in the training data (measured above); prefer the _clean sets.
• Tokenizer. 32k pieces shared with Sanskrit (in SLP1 transliteration) and math; FineWeb text takes 13% more tokens than with GPT-2's tokenizer (1.52 vs 1.35 per word).

Versions

Each version is a git tag on this repository; main is version v0.2.0. Pin one with revision=. A version is one training run, named by its run id in the training log.

Files

Attribution

The GPT-2 124M reference curve is from Keller Jordan's modded-nanogpt (MIT), record 2024-06-06, and the FineWeb validation shard is llm.c's (Karpathy), as re-hosted in kjj0/fineweb10B-gpt2.

Licence

This release is for research and non-commercial use.

• Weights (model.safetensors), tokenizer.model and tokenizer.json: CC BY-NC 4.0 (LICENSE). Non-commercial use (research, teaching, non-profit work) with attribution to "Muse Mesh Private Limited".
• Code (configuration_mume.py, modeling_mume.py, tokenization_mume.py): Apache-2.0 (LICENSE-CODE).
• Commercial use: contact kushal@muse-mesh.com.

Why non-commercial: The weights saw only permissively licensed text: FineWeb (ODC-By 1.0). The tokenizer is the issue: the shared 32k tokenizer was trained on a 1.2 GB sample of which 400 MB is Sanskrit from the Sansar corpus, and part of that Sanskrit is licensed for non-commercial use only (GRETIL, CC BY-NC-SA 4.0; Muktabodha, CC BY-NC 4.0) or carries no licence statement. The tokenizer, and the weights that only work with it, are therefore released under CC BY-NC 4.0, the same terms as the Sansar Sanskrit tokenizer. An Apache-2.0 line is planned as separate repositories: a new tokenizer trained without those sources, and the models retrained on it.

Citation

@misc{mume_english_125m_2026,
  title  = {Mume English 125M},
  author = {Muse Mesh},
  year   = {2026},
  note   = {v0.2.0},
  url    = {https://huggingface.co/MuseMesh/mume-english-125m}
}

Contact: kushal@muse-mesh.com