MiniCPM5-1B-Twi
A Twi (Akan) language adaptation of openbmb/MiniCPM5-1B,
produced by tokenizer extension → continued pretraining (CPT) → supervised fine-tuning (SFT).
The goal of this v1 is fluent Twi generation — it speaks natural, idiomatic Twi with correct
Ghanaian orthography (ɛ/ɔ).
- Base model: openbmb/MiniCPM5-1B (a 1.1B-param Llama-architecture model)
- Languages: Twi/Akan (
tw) primary; retains English - Live demo: ghananlpcommunity/MiniCPM5-1B-Twi-Chat
- Training code & write-up: https://github.com/michsethowusu/minicpm5-twi-finetune
⚠️ Important: decoding settings
Use sampling with a repetition penalty. Under greedy decoding this model degenerates into repetition loops. Recommended:
temperature=0.7, top_p=0.9, repetition_penalty=1.3, no_repeat_ngram_size=3
These are set as defaults in generation_config.json.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "ghananlpcommunity/MiniCPM5-1B-Twi"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Kyerɛ me nkyerɛkyerɛmu tiawa fa Ghana aduane ho."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
enable_thinking=False, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200, do_sample=True,
temperature=0.7, top_p=0.9, repetition_penalty=1.3, no_repeat_ngram_size=3)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
Approach
1. Tokenizer extension. The base tokenizer split Twi very inefficiently — the Ghanaian vowels ɛ (U+025B) and ɔ (U+0254) each became two byte-level tokens, giving a Twi fertility of 2.90 tokens/word vs 1.29 for English. We trained an auxiliary byte-level BPE on clean Twi and added the top 6,000 Twi word-pieces to the tokenizer (130,560 → 136,560), then resized the model embeddings and warm-started each new token from the mean of its original subword embeddings. This cut Twi fertility to ~1.26 tokens/word (−57%).
2. Continued pretraining (CPT). Full-parameter CPT for 1 epoch (~666M tokens) on a mix of 65% monolingual Twi / 20% EN–Twi parallel / 15% English replay (the English replay mitigates catastrophic forgetting at this scale). Loss 6.29 → 1.90.
3. Supervised fine-tuning (SFT). LoRA SFT (rank 32) for 3 epochs on ~200k shuffled instruction/QA examples — a blend of short factual QA and longer passage-grounded QA in Twi. Loss 4.14 → 2.30.
Training data
All Twi text was Unicode-normalized to canonical Latin ɛ/ɔ before training.
| Stage | Dataset |
|---|---|
| Tokenizer + CPT (monolingual) | ghananlpcommunity/pristine-twi |
| CPT (parallel / grounding) | ghananlpcommunity/pristine-twi-english |
| SFT (short QA) | ghananlpcommunity/gooaq-twi-2m |
| SFT (long / grounded QA) | michsethowusu/ghana-chat-corpus-ak |
Evaluation
Held-out Twi perplexity after CPT: 8.26. Qualitatively the model produces fluent, idiomatic Twi with correct orthography. Descriptive/explanatory prompts work well.
Limitations
This is a v1 focused on fluency, and it has real weaknesses:
- Factual accuracy is unreliable — output is fluent but can be wrong, and long answers sometimes drift or include invented words toward the end.
- Reasoning / math is weak — full-parameter CPT eroded some of the base model's instruction-following/reasoning ("alignment tax"), and the v1 SFT data was plain QA.
- Requires a repetition penalty (see above) — do not use greedy decoding.
- SFT data included machine-translated content, which introduces some translationese.
Planned v2 improvements: cleaner/curated SFT data, instruction-format English replay during CPT, and chain-of-thought-shaped examples (reason in English → answer in Twi).
Credits & license
Built on openbmb/MiniCPM5-1B (Apache-2.0) using datasets from the Ghana NLP community. Released under Apache-2.0. Developed by michsethowusu / Ghana NLP.