ViCLIP-OT is the first foundation vision-language model specifically designed for Vietnamese image-text retrieval.
ViCLIP-OT models combine CLIP-style contrastive learning with a novel Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance cross-modal alignment and reduce the modality gap. ViCLIP-OT achieves state-of-the-art performance on Vietnamese image-text retrieval benchmarks with strong zero-shot generalization.
Model Variants
| Variant | Contrastive Loss | OT Loss | Params | Hugging Face |
|---|---|---|---|---|
| ViCLIP-OT (you are here) | CLIP | SIGROT | 221M | minhnguyent546/ViCLIP-OT |
| ViSigLIP-OT | SigLIP | SIGROT | 221M | minhnguyent546/ViSigLIP-OT |
Quick Start
Model Overview
ViCLIP-OT is a dual-encoder vision-language model with 221M parameters. The text encoder is based on Vietnamese-SBERT, and the image encoder uses a ViT-B/16 backbone pre-trained with the DINOv3 framework. The model is trained with a hybrid objective that combines CLIP-style contrastive learning and the proposed Similarity-Graph Regularized Optimal Transport (SIGROT) loss.
| Feature | Text Encoder | Image Encoder |
|---|---|---|
| Base Model | Vietnamese-SBERT | DINOv3-ViT-B/16 |
| Parameters | 135M | 86M |
| Input Specification | 256 tokens (max) | 224 x 224 pixels |
| Pooling Strategy | Mean pooling | Global average pooling |
| Output Dimension | 768 | 768 |
Intended Uses
ViCLIP-OT is a multimodal embedding model that encodes both Vietnamese text and images into a shared representation space. It can be used for:
- Vietnamese image-text retrieval (text-to-image and image-to-text, image-to-image)
- Cross-modal semantic search in Vietnamese
- Feature extraction for downstream Vietnamese vision-language tasks (e.g., visual question answering, image classification)
Usage
pip install \
'transformers>=4.57.0,<5.0.0' \
'torch>=2.8.0,<2.10.0' \
'torchvision>=0.23.0,<0.25.0' \
timm \
pillow
from transformers import AutoModel, AutoProcessor
import torch
# Initialize the model
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')
model = AutoModel.from_pretrained('minhnguyent546/ViCLIP-OT', trust_remote_code=True)
model.to(device)
# Example images and sentences
image_uris = [
'http://images.cocodataset.org/train2014/COCO_train2014_000000138621.jpg',
'http://images.cocodataset.org/train2014/COCO_train2014_000000190580.jpg',
]
sentences = [
'Một con mèo màu trắng',
'Một con mèo màu đen',
'Một cô gái đang lướt sóng',
]
# Encode text
text_embeddings = model.encode_text(
sentences=sentences,
batch_size=32,
show_progress_bar=True,
convert_to_tensor=True,
normalize=True,
padding=True,
truncation=True,
max_length=512,
)
# Encode images
image_embeddings = model.encode_image(
images=image_uris,
batch_size=32,
show_progress_bar=True,
convert_to_tensor=True,
normalize=True,
)
# Compute cosine similarity between image and text embeddings
similarities = image_embeddings @ text_embeddings.T
print(similarities)
# tensor([[0.2438, 0.1506, 0.7248],
# [0.4299, 0.5287, 0.2329]])
pip install \
'transformers>=4.57.0,<5.0.0' \
'sentence-transformers>=4.0.0' \
'torch>=2.8.0,<2.10.0' \
'torchvision>=0.23.0,<0.25.0' \
timm \
pillow
from sentence_transformers import SentenceTransformer
# Initialize the model
model = SentenceTransformer('minhnguyent546/ViCLIP-OT', trust_remote_code=True)
# Example images and sentences
image_uris = [
'http://images.cocodataset.org/train2014/COCO_train2014_000000138621.jpg',
'http://images.cocodataset.org/train2014/COCO_train2014_000000190580.jpg',
]
sentences = [
'Một con mèo màu trắng',
'Một con mèo màu đen',
'Một cô gái đang lướt sóng',
]
# Encode text
text_embeddings = model.encode(
sentences,
batch_size=32,
show_progress_bar=True,
convert_to_tensor=True,
normalize_embeddings=True,
)
# Encode images
image_embeddings = model.encode(
image_uris,
batch_size=32,
show_progress_bar=True,
convert_to_tensor=True,
normalize_embeddings=True,
)
# Compute cosine similarity between image and text embeddings
similarities = image_embeddings @ text_embeddings.T
print(similarities)
# tensor([[0.2438, 0.1506, 0.7248],
# [0.4299, 0.5287, 0.2329]])
Training Details
ViCLIP-OT is trained on UIT-OpenViIC, a large-scale open-domain Vietnamese image captioning dataset containing 9,088 training images with approximately 42,000 captions featuring diverse real-world scenes.
For more details, please refer to the GitHub repository.
Evaluation Results
Image-Text Retrieval on UIT-OpenViIC
The table below summarizes retrieval performance on the UIT-OpenViIC test set. Both models also substantially outperform pretrained multilingual vision-language models evaluated in a zero-shot setting.
Zero-shot image–text retrieval results on KTVIC and Crossmodal-3600
The table below reports zero-shot retrieval results on KTVIC (with near-duplicate images removed against the UIT-OpenViIC training set) and Crossmodal-3600 (using Vietnamese captions).
License
This project is licensed under the GNU General Public License v3.0 (GPL-3.0).
Citation
If you find ViCLIP-OT useful in your research, please cite the following paper:
@misc{tran2026viclipotfoundationvisionlanguagemodel,
title={ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport},
author={Quoc-Khang Tran and Minh-Thien Nguyen and Nguyen-Khang Pham},
year={2026},
eprint={2602.22678},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2602.22678},
}