ViSigLIP-OT

ViCLIP-OT is the first foundation vision-language model specifically designed for Vietnamese image-text retrieval.

ViCLIP-OT models combine CLIP-style contrastive learning with a novel Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance cross-modal alignment and reduce the modality gap. ViCLIP-OT achieves state-of-the-art performance on Vietnamese image-text retrieval benchmarks with strong zero-shot generalization.

Model Variants

Quick Start

Model Overview

ViCLIP-OT is a dual-encoder vision-language model with 221M parameters. The text encoder is based on Vietnamese-SBERT, and the image encoder uses a ViT-B/16 backbone pre-trained with the DINOv3 framework. The model is trained with a hybrid objective that combines CLIP-style contrastive learning and the proposed Similarity-Graph Regularized Optimal Transport (SIGROT) loss.

Intended Uses

ViCLIP-OT is a multimodal embedding model that encodes both Vietnamese text and images into a shared representation space. It can be used for:

• Vietnamese image-text retrieval (text-to-image and image-to-text, image-to-image)
• Cross-modal semantic search in Vietnamese
• Feature extraction for downstream Vietnamese vision-language tasks (e.g., visual question answering, image classification)

Usage

pip install \
    'transformers>=4.57.0,<5.0.0' \
    'torch>=2.8.0,<2.10.0' \
    'torchvision>=0.23.0,<0.25.0' \
    timm \
    pillow

from transformers import AutoModel, AutoProcessor
import torch

# Initialize the model
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')
model = AutoModel.from_pretrained('minhnguyent546/ViSigLIP-OT', trust_remote_code=True)
model.to(device)

# Example images and sentences
image_uris = [
    'http://images.cocodataset.org/train2014/COCO_train2014_000000138621.jpg',
    'http://images.cocodataset.org/train2014/COCO_train2014_000000190580.jpg',
]
sentences = [
    'Một con mèo màu trắng',
    'Một con mèo màu đen',
    'Một cô gái đang lướt sóng',
]

# Encode text
text_embeddings = model.encode_text(
    sentences=sentences,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize=True,
    padding=True,
    truncation=True,
    max_length=512,
)

# Encode images
image_embeddings = model.encode_image(
    images=image_uris,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize=True,
)

# Compute cosine similarity between image and text embeddings
similarities = image_embeddings @ text_embeddings.T
print(similarities)

# tensor([[ 0.0677, -0.0673,  0.5587],
#         [ 0.2560,  0.4277,  0.0467]])

pip install \
    'transformers>=4.57.0,<5.0.0' \
    'sentence-transformers>=4.0.0' \
    'torch>=2.8.0,<2.10.0' \
    'torchvision>=0.23.0,<0.25.0' \
    timm \
    pillow

from sentence_transformers import SentenceTransformer

# Initialize the model
model = SentenceTransformer('minhnguyent546/ViSigLIP-OT', trust_remote_code=True)

# Example images and sentences
image_uris = [
    'http://images.cocodataset.org/train2014/COCO_train2014_000000138621.jpg',
    'http://images.cocodataset.org/train2014/COCO_train2014_000000190580.jpg',
]
sentences = [
    'Một con mèo màu trắng',
    'Một con mèo màu đen',
    'Một cô gái đang lướt sóng',
]

# Encode text
text_embeddings = model.encode(
    sentences,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize_embeddings=True,
)

# Encode images
image_embeddings = model.encode(
    image_uris,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize_embeddings=True,
)

# Compute cosine similarity between image and text embeddings
similarities = image_embeddings @ text_embeddings.T
print(similarities)

# tensor([[ 0.0677, -0.0673,  0.5587],
#         [ 0.2560,  0.4277,  0.0467]])

Training Details

ViCLIP-OT is trained on UIT-OpenViIC, a large-scale open-domain Vietnamese image captioning dataset containing 9,088 training images with approximately 42,000 captions featuring diverse real-world scenes.

For more details, please refer to the GitHub repository.

Evaluation Results

Image-Text Retrieval on UIT-OpenViIC

The table below summarizes retrieval performance on the UIT-OpenViIC test set. Both models also substantially outperform pretrained multilingual vision-language models evaluated in a zero-shot setting.

Zero-shot image–text retrieval results on KTVIC and Crossmodal-3600

The table below reports zero-shot retrieval results on KTVIC (with near-duplicate images removed against the UIT-OpenViIC training set) and Crossmodal-3600 (using Vietnamese captions).

License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0).

Citation

If you find ViCLIP-OT useful in your research, please cite the following paper:

@misc{tran2026viclipotfoundationvisionlanguagemodel,
  title={ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport}, 
  author={Quoc-Khang Tran and Minh-Thien Nguyen and Nguyen-Khang Pham},
  year={2026},
  eprint={2602.22678},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2602.22678}, 
}