Mume Math 125M

Mume Math 125M is a 134M-parameter language model trained from scratch on 1.33B tokens of mathematical web text (OpenWebMath), with the recipe, budget and tokenizer of Mume English 125M v0.2.0. It reads math far better than the English model of the same size (bits per byte on MATH test, clean subset: -62.6%), but solves almost no word problems (GSM8K 0.91%).

From Muse Mesh (Hugging Face). Part of the Muse Mesh English and Math models (collections). Every training run, including the ones not released, is logged at saansar.com, the Sansar Lab.

Model

Why this run: The English winner's recipe and budget, unchanged, on mathematical web text.

Evaluation

Metric: bits per UTF-8 byte (lower is better): the model's negative log-likelihood of a text divided by the text's UTF-8 byte count, so models with different tokenizers share a denominator. Each set's records are joined with </s> into one stream and scored teacher-forced in 1,024-token windows with stride 512 (every scored token after the first window has at least 512 tokens of context); </s> carries loss but no bytes. Scored with scripts/train/eval_bpb.py on a GPU in bf16 autocast. The sets are frozen and published as MuseMesh/mume-eval-suites.

The English model is MuseMesh/mume-english-125m v0.2.0 (EN1-full): the same architecture, recipe, tokenizer and token budget, trained on FineWeb instead of OpenWebMath.

There is no gsm8k_clean: the contamination check found no GSM8K test problem with a copied passage in the training data (0.13% of its 8-grams occur there, scattered), so gsm8k_test is already clean.

With 512-token windows (stride 256): owm_val 1.0198, gsm8k_test 1.0736, math_test 0.8650 (eval/bpb_ctx512.json).

Contamination, measured before training (lowercase word 8-grams of each set looked up in the whole training slice; a record has a copied passage when 5+ consecutive 8-grams hit): owm_val 1,578 of 2,704 documents (58%; the web repeats pages across shards), MATH test 1,246 of 5,000 problems (25%; solutions copied onto web pages), GSM8K test 0 of 1,319. The _clean sets drop every such record. On MATH the clean set moves this model by +1.7% (0.84625 -> 0.86027), so the MATH number is not memorisation. On OpenWebMath the clean subset reads +7.4% higher for this model and +1.7% for the English model, so part of the owm_val number does come from copies; quote the _clean columns. Against the English model on the clean sets: owm_val_clean -32.1%, math_test_clean -62.6%.

GSM8K exact match

3-shot, greedy: three worked GSM8K train examples ("Question: ...\nAnswer: ... #### N"), then the test question and "Answer:"; up to 200 new tokens, stopping at the next "Question:" or after the #### line. The prediction is the number after #### (else the last number); it counts when it equals the gold number. All 1,319 test problems (scripts/experts/math/gsm8k_exact.py).

These runs used the fp32 training weights (GPU, bf16 autocast). This repo's bf16 weights, same protocol and GPU: 13 / 1,319 = 0.99% (greedy decoding of a model this small flips on near-ties, so single problems change).

Low single digits is what a 125M base model scores here; the number is the family's first capability metric. A fine-tuned version on the sft branch (tag v0.1.0-sft) reaches 2.43% and shows why the score stays low (see its card).

Checked before release

• modeling_mume.py against the training code (scripts/train/model.py), fp32 on CPU, 6 inputs (one evaluation text per set, cropped to 1,024 tokens, and 1,024 random ids): with the same bf16 weights the logits are identical (max |difference| 0); against the fp32 training weights the bf16 storage moves logits by at most 0.205.
• Bits per byte with eval_bpb.py's scoring on the first 20 records of each set (each cut to 20,000 characters): 1.06580 (training checkpoint, fp32) vs 1.06587 (this repo, bf16 weights) = +0.0063%.
• The whole gsm8k_test set through this repo's model (bf16 weights, fp32 on CPU): 1.07000, against 1.07016 in the table (training checkpoint, GPU bf16 autocast), -0.015%.
• Tokenizer: AutoTokenizer(..., trust_remote_code=True) gives the training ids on 100/100 sample texts and decodes them back exactly on 100/100; on all 58,338 texts of the tokenizer's own check the same holds (see MuseMesh/mume-tokenizer-32k).
• Left-padded batches give the same logits as single sequences (max |difference| 3.1e-05).

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "MuseMesh/mume-math-125m"
tok = AutoTokenizer.from_pretrained(repo, revision="v0.1.0", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, revision="v0.1.0", trust_remote_code=True)

inputs = tok("Let $f(x) = x^2 + 1$. Then", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=100)
print(tok.decode(out[0], skip_special_tokens=True))

Needs transformers (tested with 5.18.0), torch and sentencepiece. trust_remote_code=True runs two small files from this repository, modeling_mume.py (the network; the Sansar model code with the classes renamed) and tokenization_mume.py (SentencePiece, so the ids are exactly the training ids); read them before you run them. Without trust_remote_code the tokenizer falls back to tokenizer.json, which agrees on short texts but splits a few near-tie words differently in long documents. The weights are stored in bfloat16; pass dtype=torch.float32 for fp32, usually the faster choice on a CPU. The tokenizer adds no BOS/EOS: training documents were separated by </s> only.

What this checkpoint wrote for that prompt (CPU, torch.manual_seed(0), 60 new tokens):

Let $f(x) = x^2 + 1$. Then $|f(x)| = |x|^2$. If $f(x) = x^2 + 1$, then $|f'(x)| = |x|^2 = |x|^2$. Thus $|f(x)| \leq |x^2 + 

and with greedy decoding:

Let $f(x) = x^2 + 1$. Then $f(x) = x^2 + 1$ and $f(x) = x^2 + 1$ for all $x \in \mathbb{R}$.

Now, let $f(x) = x^2 + 1$ and $f(

To score text instead of generating it, call the model with labels=input_ids; loss is the mean negative log-likelihood per token in nats.

Training data

OpenWebMath shards 4-15 of 114 (12 parquet files): 662,940 web pages with mathematical content, 816M words; 1,824 pages whose URL also occurs in shard 113 (the held-out shard) were skipped. Shards 0-3 were the tokenizer's math sample and are not in the training data.

Preparation (scripts/train/prep_data.py): Unicode NFC; documents longer than 20,000 characters cut into parts at whitespace (774,551 records); tokenized with the shared 32k tokenizer; one </s> after every record. A 0.5% validation stream by hash of each record's text (3,769 records); the rest is training: 770,782 records, 1,555,523,031 tokens, 5.12 GB (3.29 bytes per token).

Every training document is listed (source file, row, id; no text) in MuseMesh/mume-eval-suites, config training_manifests, split math_pretrain, with the parts that went to validation, so the slice can be rebuilt from the public source.

Training

Limitations

• Base model. It continues text. It has not been instruction-tuned or aligned: it does not follow instructions, answer questions or chat, and it will not refuse anything.
• It makes things up. Fluent text with invented facts, names, numbers, citations and proofs. Do not use it as a source of facts or of correct mathematics.
• Small and short. 134M parameters and a 1,024-token context. This implementation has no key/value cache, so long generations are slow (each new token re-reads the context).
• Web math. OpenWebMath is forum posts, lecture notes, blogs and Q&A pages with LaTeX; the model writes in that register, can switch into English prose, and has little arithmetic ability.
• Contamination. Some evaluation text occurs in the training data (measured above); prefer the _clean sets.
• Tokenizer. 32k pieces shared with Sanskrit (in SLP1 transliteration) and math; FineWeb text takes 13% more tokens than with GPT-2's tokenizer (1.52 vs 1.35 per word).

Versions

Each version is a git tag on this repository; main is version v0.1.0. Pin one with revision=. A version is one training run, named by its run id in the training log.

The fine-tuned model lives on its own branch, sft, so main stays the base model that the bits-per-byte numbers describe; the tag v0.1.0-sft pins it. A later fine-tune of this base will move the sft branch and get its own tag.

Files

Attribution

Licence

This release is for research and non-commercial use.

• Weights (model.safetensors), tokenizer.model and tokenizer.json: CC BY-NC 4.0 (LICENSE). Non-commercial use (research, teaching, non-profit work) with attribution to "Muse Mesh Private Limited".
• Code (configuration_mume.py, modeling_mume.py, tokenization_mume.py): Apache-2.0 (LICENSE-CODE).
• Commercial use: contact kushal@muse-mesh.com.

Why non-commercial: The weights saw only permissively licensed text: OpenWebMath (ODC-By 1.0). The tokenizer is the issue: the shared 32k tokenizer was trained on a 1.2 GB sample of which 400 MB is Sanskrit from the Sansar corpus, and part of that Sanskrit is licensed for non-commercial use only (GRETIL, CC BY-NC-SA 4.0; Muktabodha, CC BY-NC 4.0) or carries no licence statement. The tokenizer, and the weights that only work with it, are therefore released under CC BY-NC 4.0, the same terms as the Sansar Sanskrit tokenizer. An Apache-2.0 line is planned as separate repositories: a new tokenizer trained without those sources, and the models retrained on it.

Citation

@misc{mume_math_125m_2026,
  title  = {Mume Math 125M},
  author = {Muse Mesh},
  year   = {2026},
  note   = {v0.1.0},
  url    = {https://huggingface.co/MuseMesh/mume-math-125m}
}

Contact: kushal@muse-mesh.com