5CD-AI/visocial-T5-base

Overview

We trimmed vocabulary size to 50,589 and continually pretrained google/mt5-base[1] on a merged 20GB dataset, the training dataset includes:

• Crawled data (Million of comments and posts on Facebook)
• UIT data[2], which is used to pretrain uitnlp/visobert[2]
• MC4 ecommerce
• 10.7M comments on VOZ Forum from tarudesu/VOZ-HSD[7]
• 3.6M reviews from Amazon[3] translated into Vietnamese from 5CD-AI/Vietnamese-amazon_polarity-gg-translated

Here are the results on 3 downstream tasks on Vietnamese social media texts, including Hate Speech Detection(UIT-HSD), Toxic Speech Detection(ViCTSD), Hate Spans Detection(ViHOS):

Visocial-T5-base versus other T5-based models in terms of Vietnamese HSD-related task performance with Macro F1-score:

Fine-tune Configuration

We fine-tune 5CD-AI/visocial-T5-base on 3 downstream tasks with transformers library with the following configuration:

• seed: 42
• training_epochs: 4
• train_batch_size: 4
• gradient_accumulation_steps: 8
• learning_rate: 3e-4
• lr_scheduler_type: linear
• model_max_length: 256
• metric_for_best_model: eval_loss
• evaluation_strategy: steps
• eval_steps=0.1

References

[1] mT5: A massively multilingual pre-trained text-to-text transformer

[2] ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing

[3] The Amazon Polarity dataset

[4] PhoBERT: Pre-trained language models for Vietnamese

[5] Improving Sequence Tagging for Vietnamese Text Using Transformer-based Neural Models

[6] ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing

[7] ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

[8] ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation