Zomi QLoRA v1 — Zomi Language Model
A Qwen 2.5 3B fine-tuned on Zomi (Zou) language text using QLoRA (rank 128).
This model can generate and understand Zomi text. It is the first publicly available language model for the Zomi language (spoken by ~500,000 people across Myanmar, India, and Bangladesh).
What This Model Can Do
- Generate coherent Zomi text
- Complete Zomi sentences and passages
- Understand Zomi grammar, vocabulary, and structure
- Serve as a foundation for further fine-tuning
What This Model Cannot Do (Yet)
- Follow instructions or commands (Phase 2 — instruction fine-tuning — is in progress)
- Translate between Zomi and English on demand
- Hold conversations
This is a continued pretraining / language adaptation model. It speaks Zomi but does not yet follow directions. Phase 2 (instruction tuning) will add command-following ability.
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "paumkim/zomi-qlora-v1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
device_map="auto",
torch_dtype="auto",
)
# Generate Zomi text
prompt = "Pasian in hong it"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Fine-Tune It Yourself
This model is a QLoRA adapter (rank 128) on top of Qwen 2.5 3B. You can load it and fine-tune further:
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B", device_map="auto")
model = PeftModel.from_pretrained(base, model_name)
# Continue training with your own Zomi data
Training Details
| Parameter | Value |
|---|---|
| Base model | Qwen 2.5 3B |
| Method | QLoRA (rank 128, alpha 256) |
| Trainable params | 239M (7.2% of base) |
| Data | ~54,000 chunks of Zomi text from public sources |
| Hardware | NVIDIA A100 80GB |
| GPU hours | ~20 hours |
Dataset
The model was trained on a corpus of Zomi text collected from public sources including:
- Zomi literature and publications
- Community-contributed content
- Public domain religious texts in Zomi
All data was collected with respect for community ownership of the Zomi language.
Why This Model Exists
Most AI models do not support Zomi. The Zomi language has ~500,000 speakers but zero commercial AI representation. This model is an attempt to change that — built by a Zomi speaker, for Zomi speakers.
Phase 2 (Coming)
Instruction fine-tuning is in progress. The Phase 2 model will follow commands in Zomi, enabling translation,问答, content generation, and more.
License
MIT — free to use, modify, and distribute. We believe the Zomi language should belong to its community, not to any company.