5CD-AI/visobert-14gb-corpus
Overview
We continually pretrain uitnlp/visobert on a merged 14GB dataset, the training dataset includes:
- Crawled data (100M comments and 15M posts on Facebook)
- UIT data, which is used to pretrain
uitnlp/visobert - MC4 ecommerce
Here are the results on 4 downstream tasks on Vietnamese social media texts, including Emotion Recognition(UIT-VSMEC), Hate Speech Detection(UIT-HSD), Spam Reviews Detection(ViSpamReviews), Hate Speech Spans Detection(ViHOS):
Usage (HuggingFace Transformers)
Install transformers package:
pip install transformers
Then you can use this model for fill-mask task like this:
from transformers import pipeline
model_path = "5CD-AI/visobert-14gb-corpus"
mask_filler = pipeline("fill-mask", model_path)
mask_filler("shop làm ăn như cái <mask>", top_k=10)
Fine-tune Configuration
We fine-tune 5CD-AI/visobert-14gb-corpus on 4 downstream tasks with transformers library with the following configuration:
- seed: 42
- gradient_accumulation_steps: 1
- weight_decay: 0.01
- optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
- training_epochs: 30
- model_max_length: 128
- learning_rate: 1e-5
- metric_for_best_model: wf1
- strategy: epoch
And different additional configurations for each task:
| Emotion Recognition | Hate Speech Detection | Spam Reviews Detection | Hate Speech Spans Detection |
|---|---|---|---|
| - train_batch_size: 64- lr_scheduler_type: linear | - train_batch_size: 32- lr_scheduler_type: linear | - train_batch_size: 32- lr_scheduler_type: cosine | - train_batch_size: 32- lr_scheduler_type: cosine |