CLIP-ViT-Base Fusion Model for Multi-Modal Hate Speech Detection

A PyTorch-based multi-modal (image + text) hateful content classification model using CLIP encoder with late fusion architecture, trained on the MMHS150K dataset for detecting hate speech in social media memes and posts.

🎯 Model Description

This model implements a late fusion architecture with gated attention mechanism for detecting hateful content in social media memes and posts. It combines visual and textual features using OpenAI's CLIP (ViT-Base-Patch32) as the backbone encoder.

The model performs multi-label classification across 5 hate speech categories, making it capable of detecting multiple types of hate in a single post (e.g., content that is both racist and sexist).

🏗️ Architecture

┌─────────────┐     ┌─────────────┐
│   Image     │     │    Text     │
│   Encoder   │     │   Encoder   │
│ (CLIP ViT)  │     │ (CLIP Text) │
└──────┬──────┘     └──────┬──────┘
       │                   │
       ▼                   ▼
┌─────────────┐     ┌─────────────┐
│  Projection │     │  Projection │
│   (Linear)  │     │   (Linear)  │
└──────┬──────┘     └──────┬──────┘
       │                   │
       └─────────┬─────────┘
                 │
                 ▼
         ┌─────────────┐
         │ Gated Fusion│◄── Modality presence flags
         │   Module    │    (handles missing modalities)
         └──────┬──────┘
                │
                ▼
    ┌───────────────────────┐
    │  Interaction Features │
    │  • Fused embedding    │
    │  • Text embedding     │
    │  • Visual embedding   │
    │  • |text - visual|    │
    │  • text ⊙ visual      │
    └───────────┬───────────┘
                │
                ▼
         ┌─────────────┐
         │Classification│
         │  Head (MLP)  │
         │  → 5 classes │
         └─────────────┘

🔑 Key Features

🏷️ Output Classes

📊 Evaluation Results

Test Set Performance

Per-Class Performance (Validation Set)

⚙️ Optimized Thresholds

The model uses per-class calibrated thresholds for optimal performance (instead of default 0.5):

📈 Model Comparison

🎓 Training Data

MMHS150K Dataset

The model was trained on the MMHS150K (Multi-Modal Hate Speech) dataset, a large-scale multi-modal hate speech dataset collected from Twitter containing 150,000 tweet-image pairs annotated for hate speech detection.

Dataset Splits

Dataset Reference

Paper: "Exploring Hate Speech Detection in Multimodal Publications" (WACV 2020)
Authors: Raul Gomez, Jaume Gibert, Lluis Gomez, Dimosthenis Karatzas

🔧 Training Procedure

Training Configuration

# Model Configuration
backend: clip
head: fusion
encoder_name: openai/clip-vit-base-patch32
fusion_dim: 512
max_text_length: 77
freeze_text: false
freeze_image: false

# Training Configuration
num_train_epochs: 6
per_device_train_batch_size: 32
per_device_eval_batch_size: 64
gradient_accumulation_steps: 2

# Learning Rates (Differential)
lr_encoder: 1.0e-5
lr_head: 5.0e-4

# Regularization
weight_decay: 0.02
max_grad_norm: 1.0

# Scheduler
warmup_ratio: 0.05
lr_scheduler_type: cosine

# Loss
loss_type: bce
use_logit_adjustment: false

# Precision
precision: fp16

# Data Augmentation
augment: true
aug_scale_min: 0.8
aug_scale_max: 1.0
horizontal_flip: true
color_jitter: true

# Early Stopping
early_stopping_patience: 3
metric_for_best_model: roc_macro

Training Highlights

• Differential Learning Rates: Encoder (1e-5) vs Classification Head (5e-4)
• Mixed Precision: FP16 training for efficiency
• Data Augmentation: Random scaling, horizontal flip, color jitter
• Threshold Calibration: Per-class threshold optimization on validation set
• Early Stopping: Patience of 3 epochs based on ROC-AUC macro
• Best Checkpoint: Selected based on validation ROC-AUC macro score

Computational Resources

• Training Time: ~6 epochs
• Best Checkpoint: Step 33,708
• Hardware: GPU with FP16 support

🚀 How to Use

Quick Start (Recommended)

import torch
from PIL import Image
from transformers import AutoModel, CLIPProcessor

# Load model with trust_remote_code=True
model = AutoModel.from_pretrained(
    "Amirhossein75/clip-vit-base-mmhs150k-fusion",
    trust_remote_code=True
)
model.eval()

# Load CLIP processor
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Prepare inputs
image = Image.open("path/to/image.jpg").convert("RGB")
text = "sample text from the meme"

inputs = processor(
    text=[text],
    images=[image],
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=77
)

# Run inference
with torch.no_grad():
    result = model.predict(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        pixel_values=inputs["pixel_values"]
    )

print(result)
# {'predictions': {'racist': False, 'sexist': True, 'homophobe': False, 'religion': False, 'otherhate': False},
#  'probabilities': {'racist': 0.12, 'sexist': 0.78, 'homophobe': 0.05, 'religion': 0.02, 'otherhate': 0.15}}

Batch Inference

import torch
from PIL import Image
from transformers import AutoModel, CLIPProcessor

# Load model
model = AutoModel.from_pretrained(
    "Amirhossein75/clip-vit-base-mmhs150k-fusion",
    trust_remote_code=True
)
model.eval()

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Prepare batch
images = [Image.open("image1.jpg").convert("RGB"), Image.open("image2.jpg").convert("RGB")]
texts = ["text for image 1", "text for image 2"]

inputs = processor(
    text=texts,
    images=images,
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=77
)

# Get raw logits and probabilities
with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        pixel_values=inputs["pixel_values"]
    )
    probabilities = torch.sigmoid(outputs["logits"])

# Apply optimized thresholds
thresholds = torch.tensor([0.35, 0.70, 0.75, 0.30, 0.60])
predictions = (probabilities > thresholds).int()
print(predictions)

Using with GPU

import torch
from transformers import AutoModel, CLIPProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    "Amirhossein75/clip-vit-base-mmhs150k-fusion",
    trust_remote_code=True
).to(device)
model.eval()

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# ... prepare inputs ...
inputs = {k: v.to(device) for k, v in inputs.items()}

with torch.no_grad():
    result = model.predict(**inputs)

Download Config Files Only

from huggingface_hub import hf_hub_download
import json

# Download inference config
config_path = hf_hub_download(
    repo_id="Amirhossein75/clip-vit-base-mmhs150k-fusion",
    filename="inference_config.json"
)

with open(config_path) as f:
    config = json.load(f)

print(f"Classes: {config['class_names']}")
print(f"Thresholds: {config['thresholds']}")
print(f"Encoder: {config['encoder_name']}")

📁 Model Files

☁️ AWS SageMaker Deployment

This model is compatible with AWS SageMaker for cloud deployment:

from sagemaker.pytorch import PyTorchModel
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

model = PyTorchModel(
    model_data="s3://your-bucket/model.tar.gz",
    role=role,
    entry_point='inference.py',
    source_dir='sagemaker',
    framework_version='2.1.0',
    py_version='py310',
)

predictor = model.deploy(
    instance_type='ml.g4dn.xlarge',
    initial_instance_count=1,
)

# Make prediction
import base64
with open('image.jpg', 'rb') as f:
    image_b64 = base64.b64encode(f.read()).decode('utf-8')

response = predictor.predict({
    'instances': [{
        'text': 'Sample text content',
        'image_base64': image_b64,
    }]
})

See the SageMaker documentation for full deployment guide.

⚠️ Intended Uses & Limitations

✅ Intended Uses

• Content moderation for social media platforms
• Detecting hateful memes and posts
• Research in multi-modal hate speech detection
• Building content safety systems
• Pre-filtering potentially harmful content for human review

⚠️ Limitations

❌ Out-of-Scope Uses

• NOT for making final moderation decisions without human review
• NOT suitable for legal or compliance purposes without additional validation
• NOT for censorship or suppression of legitimate speech
• NOT for targeting or profiling individuals

🛡️ Ethical Considerations

• This model should be used as a tool to assist human moderators, not replace them
• False positives may incorrectly flag legitimate content
• False negatives may miss harmful content
• Regular evaluation and bias auditing is recommended
• Consider the cultural and contextual factors in deployment

📝 Citation

If you use this model, please cite:

@misc{yousefi2024multimodal,
  title={Multi-Modal Hateful Content Classification with CLIP Fusion},
  author={Yousefi, Amirhossein},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/Amirhossein75/clip-vit-base-mmhs150k-fusion}
}

Dataset Citation

@inproceedings{gomez2020exploring,
  title={Exploring Hate Speech Detection in Multimodal Publications},
  author={Gomez, Raul and Gibert, Jaume and Gomez, Lluis and Karatzas, Dimosthenis},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  pages={1470--1478},
  year={2020}
}

CLIP Citation

@inproceedings{radford2021learning,
  title={Learning Transferable Visual Models From Natural Language Supervision},
  author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and others},
  booktitle={International Conference on Machine Learning},
  pages={8748--8763},
  year={2021}
}

🔗 Links

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please see the GitHub repository for contribution guidelines.