NepaliGPT: Nepali Language Generative Pretrained Transformer Model
This is an experiment for developing a language generation model for the Nepali language. Causal Language Model which can predict the next possible tokens given a context in Nepali language.
Dataset Used
A large corpus of 9.3 GB size has been collected from different sources on the internet. The sources include
- Nepali Books found online.
- Nepali News Article from Nepali news portals.
- Nepali text collected from different open source Nepali NLP datasets.
Hyperparameters Used
Learning rate -> 2e-5
Weight Decay -> 0.01
Number of training epochs -> 5 \
bf16 -> True
Base Model Architecture -> GPT-2 \
Training Results
It achieves the following results on the evaluation set:
| Training Loss | Validation Loss | Perplexity |
|---|---|---|
| 3.3968 | 3.2705 | 26.3245 |