5CD-AI/viso-twhin-bert-large
Overview
We reduce TwHIN-BERT's vocabulary size to 20k on the UIT dataset and continue pretraining for 10 epochs.
Here are the results on 4 downstream tasks on Vietnamese social media texts, including Emotion Recognition(UIT-VSMEC), Hate Speech Detection(UIT-HSD), Spam Reviews Detection(ViSpamReviews), Hate Speech Spans Detection(ViHOS):
Usage (HuggingFace Transformers)
Install transformers package:
pip install transformers
Then you can use this model for fill-mask task like this:
from transformers import pipeline
model_path = "5CD-AI/viso-twhin-bert-large"
mask_filler = pipeline("fill-mask", model_path)
mask_filler("đúng nhận sai <mask>", top_k=10)
Fine-tune Configuration
We fine-tune 5CD-AI/viso-twhin-bert-large on 4 downstream tasks with transformer library with the following configuration:
- train_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- weight_decay: 0.01
- optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- training_epochs: 30
- model_max_length: 128
- metric_for_best_model: wf1
- strategy: epoch
And different additional configurations for each task:
| Emotion Recognition | Hate Speech Detection | Spam Reviews Detection | Hate Speech Spans Detection |
|---|---|---|---|
| - learning_rate: 1e-5 | - learning_rate: 5e-6 | - learning_rate: 1e-5 | - learning_rate: 5e-6 |