Model Card for Model ID

IRIISNEPAL/RoBERTa_Nepali_110M is a RoBERTa-based transformer model developed specifically for the Nepali language. This 110-million-parameter model is intended for tasks in natural language understanding (NLU), such as sentiment analysis, text classification, and named entity recognition in Nepali.

Model Details

Model Description

• Developed by: Institute of Research and Innovation in Intelligent Systems (IRIIS)
• Model type: RoBERTa-based transformer model specifically trained on Nepali language data
• Model Size: 110 million parameters
• Language (NLP): Nepali
• Training Objective: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)

The IRIISNEPAL/RoBERTa_Nepali_110M model aims to provide a robust tool for NLP tasks specific to the Nepali language, supporting NLP research and applications within low-resource languages.

Uses

Direct Use

The model provides contextual embeddings for each token in an input sequence (last_hidden_state) and a pooled representation of the entire input (pooler_output). These outputs can be used for:

• Text Classification: Using pooler_output to classify the overall sentiment, intent, or category of a sentence.
• Token-Level Tasks: Leveraging last_hidden_state to perform tasks like named entity recognition (NER) or part-of-speech tagging by predicting labels for individual tokens.
• Sentence Embeddings: Using pooler_output as an embedding for the entire input text for similarity search or clustering tasks.

Downstream Use

The model was evaluated on the Nepali Language Understanding Evaluation (Nep-gLUE) benchmark, demonstrating strong performance across various natural language understanding (NLU) tasks:

• Named Entity Recognition (NER): 93.74
• Part-of-Speech (POS) Tagging: 97.52
• Categorical Classification (CC): 94.68
• Categorical Pair Similarity (CPS): 96.49

These results indicate the model’s effectiveness in capturing language nuances for multiple NLU tasks in Nepali.

Bias, Risks, and Limitations

The model may exhibit biases present in its training data, especially regarding social, cultural, and regional aspects of the Nepali language. Users should exercise caution when deploying it in applications that might perpetuate stereotypes or cultural biases.

Recommendations

It’s advisable for users to monitor model outputs for fairness and avoid high-stakes applications without thorough testing. Fine-tuning or retraining may be necessary for sensitive applications.

How to Get Started with the Model

Use the code below to get started with the model.

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")
model = AutoModel.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")

text = "नेपालमा पर्यटनको विकास गर्नुपर्ने आवश्यकता छ।"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

Training Details

Training Data

The model was trained on a 27.5 GB Nepali language corpus compiled from 99 Nepali news websites. This dataset represents the largest Nepali language corpus to date, providing a significant expansion in training resources for the language. The preprocessing involved deduplication, translation/removal of non-Nepali content, and noise reduction.

You can find detailed information about the dataset in the dataset card on Hugging Face.

Training Procedure

• Training Regime: Mixed precision (fp16) on TPU v4-8 hardware
• Batch Size: 256
• Learning Rate: 1e-4 with a warmup over the first 10,000 steps followed by linear decay

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Max Sequence Length: 512 tokens
• Learning Rate Scheduler: Linear with warmup
• Optimizer: AdamW with β1 = 0.9, β2 = 0.999, and L2 weight decay of 0.01
• Dropout Probability: 0.1 across all layers
• Activation Function: GELU

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on the Nepali Language Evaluation Benchmark (Nep-gLUE), which includes tasks like Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, text classification, and categorical pair similarity.

Metrics

• Macro-F1 for the Nepali Language Understanding Evaluation (Nep-gLUE) benchmark.

Results

On Nep-gLUE, the model outperformed existing state-of-the-art models with an overall score of 95.60, reflecting its strong language understanding capabilities.

Model Examination

Performance analysis indicates robustness in capturing grammatical and syntactical features of Nepali. However, the model may have limited effectiveness in handling dialect-specific content or informal language.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications

Model Architecture and Objective

RoBERTa architecture with 12 transformer layers, hidden size of 768, 12 attention heads, and 110 million parameters. This architecture facilitates strong bidirectional attention for accurate language understanding.

Compute Infrastructure

• Hardware: TPU v4-8 and Nvidia GeForce RTX 3090 GPUs
• Software: Python, PyTorch, Hugging Face Transformers

Citation

@misc{thapa2024developmentpretrainedtransformerbasedmodels,
      title={Development of Pre-Trained Transformer-based Models for the Nepali Language}, 
      author={Prajwal Thapa and Jinu Nyachhyon and Mridul Sharma and Bal Krishna Bal},
      year={2024},
      eprint={2411.15734},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.15734}, 
}

Model Card Contact

For questions and support, contact IRIIS Nepal.