Gemma-3-4B — Urdu Education & Reasoning
Submission to the Adaption Labs AutoScientist Challenge (Urdu language track). This is a Gemma-3-4B model adapted for Urdu. The training data was produced by translating and localising English knowledge corpora into Urdu with Adaption AutoScientist and the Adaptive Data pipeline. The model is evaluated on UrduMMLU, a 26,431-question benchmark written natively in Urdu.
Live demo: https://huggingface.co/spaces/abdullah693/urdu-edu-reasoning
Summary of results
On UrduMMLU (Urdu prompt, zero-shot), the model scores 46.21%, compared with 44.96% for the
base gemma-3-4b-it. Accuracy improves in every domain that carries transferable knowledge (STEM,
professional, social sciences, general knowledge). The only domain that does not improve is Urdu
literature, which is discussed under Limitations.
What this entry validates
The challenge asks whether Adaption's adaptive-data tooling can build useful, locally-relevant models. We tested a specific claim: adapting English knowledge corpora into Urdu with AutoScientist produces measurable gains on a native Urdu benchmark for knowledge that is language-independent. The result supports the claim and also marks its boundary.
- Every transferable domain improved (STEM +5.9, professional +3.6, other +3.6, social sciences +2.7), and the model exceeded its base overall (+1.33).
- The effect does not extend to language-intrinsic content. Urdu literature declined by 2.5 points, because that knowledge cannot be obtained by translating English sources and requires native Urdu data.
The practical conclusion for low-resource adaptation: AutoScientist-based translation is effective for the science, mathematics, reasoning, and social-knowledge portions of a benchmark, and should be paired with native-language collection for culturally specific content.
Per-domain results
| Domain | Base Gemma-3-4B | This model | Change |
|---|---|---|---|
| STEM | 46.9 | 52.8 | +5.9 |
| Professional | 49.1 | 52.7 | +3.6 |
| Other | 42.9 | 46.5 | +3.6 |
| Social sciences | 48.7 | 51.4 | +2.7 |
| Urdu literature / Humanities | 41.3 | 38.8 | −2.5 |
| Overall | 44.96 | 46.21 | +1.33 |
Overall accuracy was measured on the full 26,431-question test set with zero unparsed responses. Per-domain base figures are from a 1,499-question stratified sample.
Method
- Source assembly. About 40,000 examples were drawn from open English datasets that cover the UrduMMLU subject areas (MMLU, GSM8K, MATH, ARC, AQuA, CommonsenseQA) together with native-Urdu instruction, grammar, and literature data. Training splits only.
- Adaptation (Adaptive Data + AutoScientist). English rows were translated and localised into Pakistani Urdu. The adaptation step also produced a reformulated prompt, an answer with explanation, and an English reasoning trace for each row.
- Fine-tuning. Gemma-3-4B was supervised-fine-tuned on the adapted set.
- Evaluation. Generation with answer parsing on UrduMMLU, Urdu prompt, zero-shot.
Training data: abdullah693/adaption-urdu-edu-cultural-reasoning (~39,913 examples).
Reproducibility
- Harness validation. Our evaluation reproduces the base
gemma-3-4b-itat 44.96%, within 0.1 point of the 44.88% reported in the UrduMMLU paper, which confirms the protocol matches. - Contamination. UrduMMLU is the held-out benchmark and was not used in training.
- Protocol. Urdu prompt template from the UrduMMLU repository, generation plus answer parsing, zero-shot, full test set.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
mid = "abdullah693/gemma-3-4b-it-urdu-edu-reasoning"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "نظامِ شمسی میں کتنے سیارے ہیں؟"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[1]:], skip_special_tokens=True))
Limitations
- Urdu literature and other language-intrinsic content are weaker than the base model; native Urdu literary data is required to address this.
- At 4 billion parameters the model has limited factual depth and can produce incorrect answers, particularly on long-tail facts.
- It inherits the biases and knowledge cutoff of Gemma-3.
- Intended for research and education. Not a reliable source for examinations, religious rulings, or legal and medical advice.
Citation
@misc{gemma3_4b_urdu_edu_2026,
title = {Gemma-3-4B Urdu Education and Reasoning},
author = {abdullah693},
year = {2026},
note = {Adapted with Adaption AutoScientist; evaluated on UrduMMLU},
url = {https://huggingface.co/abdullah693/gemma-3-4b-it-urdu-edu-reasoning}
}