See our collection for all versions of MetaCLIP 2.

Run MetaCLIP 2 with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/metaclip2_mt5_worldwide_m16_224

Paper: MetaCLIP 2: A Worldwide Scaling Recipe (arXiv:2507.22062) · HF Papers

MetaCLIP 2 is a CLIP-architecture dual encoder trained on a worldwide, multi-language data distribution (not an English-filtered crawl). Worldwide variants use a 901k-token XLM-RoBERTa vocabulary instead of CLIP's English BPE; mt5 variants use SentencePiece. Class prompts can be non-English without a separate multilingual checkpoint.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of facebook/metaclip-2-mt5-worldwide-m16 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a zero-shot image-text checkpoint (MetaClip2ZeroShotClassify). The quick start below scores the same concept in English / French / Spanish / German against one distractor.

✨ Quick start (multilingual zero-shot)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import keras
from zeromodels.models.metaclip2 import (
    MetaClip2Processor,
    MetaClip2ZeroShotClassify,
)

processor = MetaClip2Processor.from_weights("zeromodels/metaclip2_mt5_worldwide_m16_224")
model = MetaClip2ZeroShotClassify.from_weights("zeromodels/metaclip2_mt5_worldwide_m16_224")

# Same concept in four languages + one English distractor.
# Unlike CLIP (English BPE) / SigLIP v1, MetaCLIP 2 is trained worldwide.
labels = [
    "a photo of teddy bears",  # English
    "une photo d'ours en peluche",  # French
    "una foto de ositos de peluche",  # Spanish
    "ein Foto von Teddybären",  # German
    "a photo of a truck",  # distractor
]
inputs = processor(text=labels, image_paths="your_image.jpg")
output = model(
    {
        "images": inputs["images"],
        "token_ids": inputs["token_ids"],
        "padding_mask": inputs["padding_mask"],
    }
)
probs = keras.ops.convert_to_numpy(
    keras.ops.softmax(output["image_logits"], axis=-1)
).squeeze()
for label, p in zip(labels, probs):
    print(f"{p:.6f}  {label}")

Load any MetaCLIP 2 variant the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• Prefer Processor.from_weights(...) so image size and tokenizer match the variant.
• Processor returns token_ids / padding_mask (not input_ids / attention_mask).
• Prompts can be non-English; keep accents exact (e.g. Teddybären vs Teddybaren).
• Some large variants may convert on the fly from upstream if no prebuilt zeromodels weights are present.
• See MetaCLIP 2 docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. MetaClip2ZeroShotClassify.from_weights("hf:facebook/metaclip-2-mt5-worldwide-m16").

Special Thanks

A huge thank you to the Meta MetaCLIP 2 authors for creating and releasing these models.

License: CC-BY-NC-4.0 (non-commercial).