BamiBERT Mixture of Experts (MoE) for Vietnamese Sentiment Analysis

Architechture systems and our ideas 👉 Paper: Doi : 10.18653/v1/2026.semeval-1.25

Demo

You can try Demo at here: https://huggingface.co/spaces/TheSon2202/bamibert-moe-vietnamese-sentiment-analytst

Overview

bamibert-moe-sentiment is a high-performance, fine-tuned text classification model tailored for Vietnamese Sentiment Analysis (e.g., classifying text into Positive or Negative sentiments).

The model is built on top of the BamiBERT architecture (an optimized BamiBERT-based model for Vietnamese) and integrated with a Mixture of Experts (MoE) gating mechanism. By utilizing sparse dynamic expert routing, the architecture maximizes representation capacity while keeping inference computationally efficient.

Key Features

• Base Architecture: Fine-tuned from Qualcomm-AI-Research/BamiBERT.
• MoE Integration: Employs a custom Mixture of Experts layer to dynamically route token representations through specialised expert pathways.
• Training Dataset: Fine-tuned on the benchmark AIVN 2019 Sentiment Analysis dataset, allowing the model to excel at understanding nuanced feedback, slang, and context-dependent sentiments in Vietnamese.
• Domain Adaptation: Optimized specifically for understanding contextual structures common in Vietnamese social media, product reviews, and customer feedback.
• Tokenization Pipeline: Designed to work hand-in-hand with VnCoreNLP for accurate Vietnamese word segmentation before token mapping.

Evaluation Results

The model performance is evaluated using 5-Fold Cross-Validation on the AIVN 2019 validation splits, along with the final inference evaluation on the Public Test set. Below is the comprehensive breakdown of our fine-tuned Mixture of Experts architecture under a Full Training (Finetune All Layers) setup:

Performance Summary

Finetune Performance

5-Fold Cross-Validation Breakdown

During full-parameter fine-tuning, the custom MoEClassifier dynamically routed token representations through \(K=2\) selected experts out of \(N=4\) available experts per forward pass. The Macro F1-Score tracking across each individual validation fold is recorded as follows:

• Fold 1: 0.9085
• Fold 2: 0.9165
• Fold 3: 0.9129
• Fold 4: 0.9035
• Fold 5: 0.9008
• ➔ Overall Mean F1-Score: 0.9045

Figure 1: Training performance metrics during the fine-tuning process.

Usage Guide

Below is a complete guide on how to load and use the BamiBertMoePredictor wrapper for inference on sentiment analysis tasks.

Prerequisites

Ensure you have installed the required dependencies and downloaded the necessary tools (e.g., VnCoreNLP) or if you cannot use this tool, you could use pyvi.

!pip install torch transformers huggingface_hub safetensors

import os
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoConfig
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from vncorenlp import VnCoreNLP

class BamiBertMoePredictor:
    """
    Production-ready Wrapper for the BamiBERT-MoE Sentiment Analysis network.
    Dynamically loads configuration properties from Hugging Face Hub and 
    handles end-to-end inference processing.
    """
    def __init__(self, repo_id: str, vncorenlp_path: str, device: str = 'cuda'):
        """
        Initializes tokenizer, remote configurations, custom MoE layers, and weights.
        
        Args:
            repo_id (str): Hugging Face Repository identifier.
            vncorenlp_path (str): File path to the VnCoreNLP compiled .jar tool.
            device (str): Execution hardware allocation ('cuda' or 'cpu').
        """
        self.device = torch.device(device if torch.cuda.is_available() else 'cpu')
        
        print(f"Fetching Tokenizer & Config from Hub: {repo_id}...")
        self.tokenizer = AutoTokenizer.from_pretrained(repo_id)
        hf_config = AutoConfig.from_pretrained(repo_id)
        
        # Inject dynamic hyperparameter definitions from remote config 
        self.config = hf_config
        self.config.model_name = getattr(hf_config, "_name_or_path", "Qualcomm-AI-Research/BamiBERT")
        self.config.max_len = getattr(hf_config, "max_len", 128)
        self.config.num_classes = getattr(hf_config, "num_classes", 2)
        self.config.num_experts = getattr(hf_config, "num_experts", 4)
        self.config.selection_k = getattr(hf_config, "selection_k", 2)
        self.config.vncorenlp_path = vncorenlp_path
        
        print("Initializing MoEBanhMiBert architecture...")
        self.model = MoEBanhMiBert(self.config)
        
        print("Pulling model.safetensors artifacts from Hub...")
        weights_path = hf_hub_download(repo_id=repo_id, filename="model.safetensors")
        
        print("➔ Injecting safe weights into model structure...")
        safetensors_weights = load_file(weights_path)
        self.model.load_state_dict(safetensors_weights)
        self.model.to(self.device)
        self.model.eval()
        
        print("Spawning VnCoreNLP Word Segmenter token pipeline...")
        self.segmenter = VnCoreNLP(self.config.vncorenlp_path, annotators="wseg")
        print("ARTIFACT LOADING COMPLETE! Predictor framework ready.")
        
    def predict(self, text: str) -> dict:
        """
        Executes end-to-end sentiment tracking over raw input strings.
        
        Args:
            text (str): Raw unstructured text query.
            
        Returns:
            dict: Structured outcomes containing mapped label and confidence rate.
        """
        # Execute text normalization mapping and syllable compound splitting
        cleaned = clean_text(text)
        word_segment = self.segmenter.tokenize(cleaned)
        segmented = " ".join(w for sentence in word_segment for w in sentence)
        
        # Generate model input tensors matching specific maximum limits
        inputs = self.tokenizer(
            segmented, 
            return_tensors="pt", 
            padding="max_length",
            truncation=True, 
            max_length=self.config.max_len
        )
        inputs = {k: v.to(self.device) for k, v in inputs.items()}
        
        # Run isolated forward execution layer
        with torch.no_grad():
            outputs = self.model(**inputs)
            logits = outputs[0] if isinstance(outputs, tuple) else outputs
            probs = F.softmax(logits, dim=-1)
            prediction = torch.argmax(probs, dim=-1).item()
            confidence = probs[0][prediction].item()
            
        label_map = {0: "Negative", 1: "Positive"}
        return {
            "text": text,
            "label": label_map[prediction],
            "confidence": f"{confidence * 100:.2f}%"
        }

# --- STANDALONE MODEL EXECUTION FLOW ---
if __name__ == "__main__":
    REPO_ID = "TheSon2202/bamibert-moe-sentiment"
    VNCORENLP_PATH = "VnCoreNLP/VnCoreNLP-1.1.1.jar"
    
    # Instantiate clean predictor sequence from shared storage weights
    predictor = BamiBertMoePredictor(repo_id=REPO_ID, vncorenlp_path=VNCORENLP_PATH)
    
    print("\n" + "="*60)
    print("RUNNING MOE SENTIMENT INFERENCE BENCHMARK")
    print("="*60)
    
    # Evaluated test sample collection (1 Positive, 1 Negative)
    test_cases = [
        "Sản phẩm dùng đỉnh thực sự, giao hàng siêu nhanh, đóng gói rất cẩn thận!",
        "đồ ăn ở đây ngon quá đi, nhưng dịch vụ tệ quá, cho 1 sao"
    ]
    
    for text in test_cases:
        res = predictor.predict(text)
        print(f"Raw Comment : {res['text']}")
        print(f"Predicted   : {res['label']} ({res['confidence']})")
        print("-" * 60)

Expected Output 1

{
  "text": "Đồ ở shop này xài bao ngon luôn á, chấm 10 điểm!",
  "label": "Positive",
  "confidence": "69.13%"
}

and Expected Output 2

{
 "text": 'đồ ăn ở đây ngon quá đi, nhưng dịch vụ tệ quá, cho 1 sao',
 "label": 'Negative',
 "confidence": '61.16%'
}

Acknowledgements & Citation

This model is fine-tuned based on the BamiBERT architecture. I would like to express my gratitude to the original authors for their foundational work.

BibTeX:

@article{BamiBERT,
  title    = {{BamiBERT: A New BERT-based Language Model for Vietnamese}},
  author   = {Dat Quoc Nguyen and Thinh Pham and Chi Tran and Linh The Nguyen},
  journal  = {arXiv preprint},
  volume   = {arXiv:2607.02259},
  year     = {2026}
}