BabyLM Arabic — Best Models per Dataset
Six GPT-2-style causal LMs (~17M params: n_embd=512, n_layer=4, n_head=8, n_ctx=512),
each the best-performing tokeniser/seed combination for one of six ~8.3M-word Arabic
corpus compositions, trained as part of an MSc dissertation on tokenisation and dataset
composition for small-scale Arabic language models (University of Stirling). See the
companion dataset cards for the five modified corpora (e.g.
kabaros/ara-ebooks-heavy-8.3M) for composition details.
Grammaticality is measured via MultiBLiMP (minimal-pair judgment, mean of 3 seeds reported in the dissertation; the score below is this specific published checkpoint's individual run, the best of the 3 seeds trained for that dataset).
| Dataset | Model dir | Tokeniser | Seed | MultiBLiMP |
|---|---|---|---|---|
Original Data (babylm-ara) | ar-camel-with-unigram-babylm-ara-seed43 | CAMeL + Unigram | 43 | 77.20 |
| 8.3M ebooks-heavy | ar-camel-custom-8.3M-wiki_0-movies_0-habibi_0-ebooks_100-seed44 | CAMeL (BPE) | 44 | 80.33 |
| 8.3M ebooks w/ 5% habibi | ar-camel-with-unigram-custom-8.3M-wiki_0-movies_0-habibi_5-ebooks_95-seed43 | CAMeL + Unigram | 43 | 79.26 |
| 8.3M ebooks w/ 5% movies | ar-camel-with-unigram-custom-8.3M-wiki_0-movies_5-habibi_0-ebooks_95 | CAMeL + Unigram | 42 (default) | 79.67 |
| 8.3M Impossible Man | ar-farasa-with-unigram-custom-impossible-man-8.3M-fixed-20%-ebooks-42%-hindawi-38% | Farasa + Unigram | 42 (default) | 79.26 |
| 8.3M Diverse | ar-camel-with-unigram-diversity-8.3M-seed44 | CAMeL + Unigram | 44 | 77.61 |
For context: the BabyLM-community/babylm-ara paper baseline (BPE, single run) scores
75.9; OLMo-2 (32B params, 6T tokens) scores 87% on the same task in Arabic.
Loading: the CAMeL/Farasa models use a custom PreTrainedTokenizerFast subclass
(camel_tokenizer.py/farasa_tokenizer.py, included in each model's files) that applies
morphological segmentation before every encode call. Load with
trust_remote_code=True:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("kabaros/<repo-name>", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("kabaros/<repo-name>")