clip-japanese-base-v2

This is a Japanese CLIP (Contrastive Language-Image Pre-training) model developed by LY Corporation. This model is an updated version of line-corporation/clip-japanese-base. It increases the training data to approximately ~2B image–text pairs and applies model distillation to improve overall performance.

How to use

1. Install packages

pip install pillow requests sentencepiece transformers torch timm

2. Run

import io
import requests
from PIL import Image
import torch
from transformers import AutoImageProcessor, AutoModel, AutoTokenizer

HF_MODEL_PATH = 'line-corporation/clip-japanese-base-v2'
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(HF_MODEL_PATH, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(HF_MODEL_PATH, trust_remote_code=True)
model = AutoModel.from_pretrained(HF_MODEL_PATH, trust_remote_code=True).to(device)

image = Image.open(io.BytesIO(requests.get('https://images.pexels.com/photos/2253275/pexels-photo-2253275.jpeg?auto=compress&cs=tinysrgb&dpr=3&h=750&w=1260').content))
image = processor(image, return_tensors="pt").to(device)
text = tokenizer(["犬", "猫", "象"]).to(device)

with torch.no_grad():
    image_features = model.get_image_features(**image)
    text_features = model.get_text_features(**text)
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
# [[1., 0., 0.]]

Model architecture

The model uses an Eva02-B Transformer architecture as the image encoder and a 12-layer BERT as the text encoder. The text encoder was initialized from rinna/japanese-clip-vit-b-16.

Evaluation

Dataset

• ImageNet-1k for image classification.
• Recruit Datasets for image classification.
• WAON for image classification.
• STAIR Captions (v2014 val set of MSCOCO) for image-to-text (i2t) and text-to-image (t2i) retrieval. We measure performance using R@1, which is the average recall of i2t and t2i retrieval.

Result

Licenses

The Apache License, Version 2.0

Citation

@misc{clip-japanese-base-v2,
    title = {CLIP Japanese Base V2},
    author={Shuntaro Okada, Shuhei Yokoo, Kei Mukaiyama, Peifei Zhu and Shuhei Nishimura}
    url = {https://huggingface.co/line-corporation/clip-japanese-base-v2},
}