Mongolian Chat Model v14.7 — Gemma-3-4B, continued pre-training + instruction tuning
Part of an effort to build open Mongolian language models that genuinely understand and answer in Mongolian.
A 4B-parameter Mongolian (Cyrillic) chat model: continued pre-training of google/gemma-3-4b-pt on a
hand-built Mongolian corpus, then instruction tuning with an emphasis on grounded answering — using
retrieved sources, citing them, and saying "I don't know" when the sources do not contain the answer.
Version numbering. v14.7 = corpus/pre-training generation 14, instruction-tuning iteration 7.
The major number changes when the pre-training corpus is rebuilt; the minor number with each instruction-tuning
run. This model is a different lineage from the earlier Llama-3.2-3B-Instruct-Mongolian release, not an
update to it.
What is different about this model
Most Mongolian models are fine-tuned on publicly packaged Mongolian datasets. I did not use them. Examined individually, they were unsuitable for this purpose for reasons that can be checked:
Billyyy/cleaned-mongolian-dataset— lowercase-normalised and stored pre-tokenised, so sentence case and proper-noun capitalisation are gone. Mongolian Cyrillic needs both.mongolian-pretrain-dataset— traditional Mongolian script (Unicode Mongolian block), not Cyrillic. Correctly excluded for a Cyrillic model rather than a quality judgement.- Others were machine-translated, duplicated web scrapes, or stored only as token ids.
Instead the corpus was collected, cleaned, and reviewed by hand, by one person, over multiple iterations:
- sources selected individually, not pulled in bulk from dataset aggregators;
- scanned material processed with OCR and re-processed where the first pass was poor;
- per-source filters written after reading samples from that source, not applied globally;
- quality problems found by reading the data, then fixed at the source and re-built;
- synthetic augmentation used only where it could be verified against the underlying text, with entries that failed verification discarded rather than kept.
That work is the substance of this model. The architecture and training recipe are standard.
Corpus composition
| category | share of tokens |
|---|---|
| academic (theses, scholarly books) | ~48% |
| news | ~25% |
| books | ~20% |
| English text | ~5.5% |
| encyclopedic | ~1% |
| verified synthetic augmentation | <1% |
| reference (organisation names, structures, key facts) | <1% |
~2.76B tokens · ~772,000 documents · 25 individually curated sources. Roughly 89% Mongolian Cyrillic, 11% English. A large share is scanned material recovered with OCR, which is why most of the effort went into cleaning rather than collection.
Instruction data
~52,000 instruction rows, also hand-built and repeatedly audited. Besides ordinary chat, mathematics, translation and dictionary work, it explicitly teaches retrieval behaviour:
- answer from the provided sources and cite them as [1], [2], [3];
- prefer the source over memory when a document contradicts what the model learned;
- abstain when the sources are about a different entity, or are on-topic but do not contain the asked detail — this last case is the one models most often get wrong, inventing a fact and citing a source for it;
- ignore the sources for chat, arithmetic, translation and writing tasks;
- say plainly that it has no real-time information when asked about current events, weather or prices.
Evaluation
Measured against the previous instruction-tuning iteration (v14.6) on held-out questions, by blind pairwise comparison with answers anonymised, order randomised and citation markers stripped:
| judging run | result |
|---|---|
| initial | v14.7 130 – v14.6 99, net +31, z 2.05 |
| re-judged with a judge instructed that answer length is not a merit | v14.7 109 – v14.6 49, 193 ties, net +60, z 4.77 |
The second run matters: the automatic judge was found to favour longer answers, so the comparison was repeated with that corrected. The result held and strengthened. The gain is concentrated in closed-book answers.
The trained retrieval behaviours — citing a source, preferring it over memory, abstaining when it does not contain the answer — were additionally checked on 491 held-out items with rule-based scoring. Those results are not quoted here: the rules reward producing the expected format as well as the right content, and an audit found much of the apparent gap was the earlier version not emitting the trained citation format at all, plus some memorisation of a single abstention sentence. The pairwise comparison above is the trustworthy evidence.
Scores from different judging runs are not comparable with each other; each table above comes from a single same-day comparison with the baseline re-judged alongside.
Evaluation method. Questions come from an append-only pool of ~1,300 real user questions, with frozen ID-only judging slices so historical scores stay reconstructible. Judging is blind and order-counterbalanced, with deliberately degraded answers mixed in as calibration checks on the judge. An audit confirmed the primary judging slice is free of training-set leakage; three secondary slices were found to be contaminated by corrective rows written from observed failures, and were retired or are reported with the clean subset separated.
Later iterations. Two further instruction-tuning runs (v14.8, v14.9) were trained and evaluated against this one. Neither measured better, so v14.7 remains the released version. Measurements that showed no improvement are reported as such.
Limitations — please read
- Single-turn only. The model is trained and served on one question at a time: it does not see earlier messages and cannot hold a conversation or follow up on its own previous answer. This follows from the instruction data being single-turn and from the 1,280-token training window, both chosen to fit the available hardware.
- Arithmetic is weak. On 1,318 held-out grade-school word problems it reaches the correct final answer about 29% of the time, and roughly 10% of the calculation steps it writes out are wrong. Accuracy falls sharply with the number of reasoning steps required — about 44% at two steps, under 10% at five or more.
- It still invents facts. A two-pass audit of 364 live answers found factual errors in roughly a quarter of all answers, and about 31% of those making a checkable claim. The first audit pass itself missed about one error in six, so this is a corrected estimate rather than a floor. It is markedly more reliable with retrieval, and worst on arithmetic and unit conversion, invented people and works, astronomy, and confabulated superlatives.
- It sometimes attaches a fact to the wrong entity, or mangles a correct one. Investigation of specific wrong answers found the underlying training data was correct in each case — the model had the right fact and bound it to the wrong subject, or dropped part of a correctly formatted answer. Phrasing matters more than it should: the same fact can come out right and wrong in one session depending on how the question is worded.
- Asked for a detail the retrieved sources do not contain, it often supplies one anyway and attaches a citation. Improved but not solved.
- It over-refuses on wording. Asked about "latest" developments it may decline even when the user explicitly asks what it already knows; removing that word produces a good answer.
- It overstates its own capabilities. Asked whether it can accept a file or an image, it may say yes. It cannot: it takes text only.
- English technical vocabulary (networking terms and similar) is thinner than general Mongolian.
- Trained for Mongolian Cyrillic. Not trained for traditional Mongolian script.
- No safety alignment beyond targeted training examples. Not suitable for unsupervised use with vulnerable users, and not a medical, legal or financial adviser.
Two further training iterations after this one produced no measurable improvement on either general chat or held-out mathematics, so these limitations should be read as the current plateau of this approach at this model size, not as a snapshot about to be superseded.
Try it
A hosted Mongolian chat interface runs at https://ai.mairi.mn/ — no installation, no sign-in.
The demo tracks the current production deployment and may be running a newer model than the weights described here. The serving layer also adds things that are not part of the model: hybrid retrieval over the training corpus, exact arithmetic for bare expressions, and a safety filter. It reports the exact model and server build it is running, so answers you see there will not always match this card.
Intended use
Mongolian question answering and chat, especially with a retrieval layer supplying source passages. Research and educational use. Not for high-stakes decisions.
Prompt format
Standard Gemma-3 chat template. For grounded answering, pass the sources in the user turn as a numbered list and ask the question after them; the model is trained to cite [n] and to abstain when the answer is absent.
Use these decoding settings. They are not cosmetic: sampling and a repetition penalty above 1.0 were both measured to corrupt thin-margin facts in this model, and the n-gram ban must exempt the prompt or the model mis-spells proper nouns the user typed.
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "munkhbayar-batkhuu/gemma-3-4b-mongolian"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "Монголын нийслэл хаана вэ"}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(
input_ids=enc["input_ids"].to(model.device),
max_new_tokens=300,
do_sample=False, # greedy — sampling degrades factual accuracy here
repetition_penalty=1.0, # must stay 1.0
no_repeat_ngram_size=6,
eos_token_id=[1, 106], # Gemma stop tokens
)
print(tok.decode(out[0][enc["input_ids"].shape[-1]:], skip_special_tokens=True))
Answers are markedly more reliable when real source passages are supplied in the prompt than when the model answers from memory alone. See the limitations above before relying on a bare prompt.
Hardware and support
Corpus building, pre-training, instruction tuning and evaluation were all done on three consumer 12 GB GPUs — the 2.76-billion-token pre-training run took 33 days of continuous compute. That constraint sets the model size, the 1,280-token training window and the single-turn design.
Support this work. We are looking for GPU hardware donations: one or more 48–80 GB GPUs. Used or previous-generation cards (A6000, A100 40/80 GB) are equally welcome. Server capacity is already in place to host up to six GPUs. With more GPU memory the next versions could:
- train with a longer context (4k–8k tokens) and support multi-turn chat;
- scale beyond 4B parameters, which measurement suggests is where the remaining ceiling lies — two successive iterations of instruction-data improvement at 4B produced no measurable gain on either general chat or held-out mathematics.
Donations go to the Mongolian Artificial Intelligence Research Institute (MAIRI, NGO), working in partnership with the Mongolian University of Life Sciences. Donors will be acknowledged on this card and at https://mairi.mn.
Contact: munkhbayar.b@muls.edu.mn · https://www.linkedin.com/in/munkhbayar-batkhuu
Licence and provenance
Derived from google/gemma-3-4b-pt and distributed under the Gemma Terms of Use, which apply to this
model as a derivative. Users must comply with the Gemma Prohibited Use Policy. The weights here are the
result of continued pre-training and instruction tuning of that base; the base model's own licence terms and
any restrictions on its use are unchanged by this work.
Author
Munkhbayar Batkhuu (BMR) — corpus construction, curation, training and evaluation.