AI-translator-eng-to-9ja
This model is a 418 Million parameter translation model, built for translating from English into Yoruba, Igbo, and Hausa. It was trained on a dataset consisting of 1,500,000 sentences (500,000 for each language), providing high-quality translations for these languages. It was built with the intention of building a system that makes it easier to communicate with LLMs using Igbo, Hausa and Yoruba languages.
Model Details
- Languages Supported:
- Source Language: English
- Target Languages: Yoruba, Igbo, Hausa
Model Usage
To use this model for translation tasks, you can load it from Hugging Face’s transformers library:
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import huggingface_hub
huggingface_hub.login()
# Load the fine-tuned model
model = M2M100ForConditionalGeneration.from_pretrained("HelpMumHQ/AI-translator-eng-to-9ja")
tokenizer = M2M100Tokenizer.from_pretrained("HelpMumHQ/AI-translator-eng-to-9ja")
# translate English to Igbo
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "ig"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("ig"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# translate English to yoruba
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions. Effective healthcare systems aim to improve health outcomes, reduce the incidence of illness, and ensure that individuals have access to necessary medical services."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "yo"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("yo"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# translate English to Hausa
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions. Effective healthcare systems aim to improve health outcomes, reduce the incidence of illness, and ensure that individuals have access to necessary medical services."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "ha"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("ha"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
Supported Language Codes
- English:
en - Yoruba:
yo - Igbo:
ig - Hausa:
ha
Training Dataset
The training dataset consists of 1,500,000 translation pairs, sourced from a combination of open-source parallel corpora and curated datasets specific to Yoruba, Igbo, and Hausa
Limitations
- While the model performs well across English-to-Yoruba, Igbo, and Hausa translations, performance may vary depending on the complexity and domain of the text.
- Translation quality may decrease for extremely long sentences or ambiguous contexts.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
Framework versions
- Transformers 4.44.2
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1