gemma-3-4b-uk-matt
google/gemma-3-4b-pt transferred to the Ukrainian-centric Lapa tokenizer with MATT (Model-Aware Tokenizer Transfer). The tokenizer keeps the original Gemma vocabulary and adds Ukrainian tokens, cutting the token count on Ukrainian text by roughly a third. Only the input embeddings differ from the original model.
Recipe
- Embeddings initialized with FOCUS (Goader/gemma-3-4b-pt-focus),
then trained with the AIM objective against the frozen original model: layers 1–16 of 34,
all_words_last_tokenalignment, MSE loss, lr 1e-4, original-vocabulary embeddings frozen. - One pass over 1.03M Ukrainian documents (Kobza corpus, 256-token windows), ~6 h on one H100.
- Weights stored in bfloat16 with the full 262,208-row embedding table.
Results (Ukrainian, 3-shot, greedy, base-model prompts)
| metric | gemma-3-4b-pt (original tokenizer) | FOCUS init | this model |
|---|---|---|---|
| Belebele uk (acc) | 72.6 | 34.6 | 68.2 |
| Global MMLU uk (acc) | 51.0 | 28.7 | 47.8 |
| FLORES en→uk (BLEU / chrF) | 26.8 / 57.2 | 9.1 / 32.0 | 25.7 / 56.2 |
| FLORES uk→en (BLEU / chrF) | 42.0 / 67.9 | 5.1 / 27.0 | 40.3 / 66.6 |
| WMT en→uk (BLEU) | 5.1 | 1.6 | 4.8 |
| XLSum uk (ROUGE-1) | 6.5 | 1.5 | 5.8 |
The model recovers 94–98 % of the original on every task while reading Ukrainian with ~35 % fewer tokens.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Goader/gemma-3-4b-uk-matt")
model = AutoModelForCausalLM.from_pretrained("Goader/gemma-3-4b-uk-matt", torch_dtype="bfloat16")
This is a base (pretrained) model, not instruction-tuned.