radio-clip_v1.0

A vision-language model pairing NVIDIA C-RADIOv3-H as the image encoder with the SigLIP2-g text tower, fine-tuned with NVIDIA TAO Toolkit using a SigLIP (sigmoid pairwise) contrastive objective. The image tower exposes a siglip2-g adaptor head that projects RADIO features into SigLIP2-g's 1536-dim joint embedding space, so image and text embeddings are directly comparable.

• Image encoder: C-RADIOv3-H (ViT-H/16, embed 1280, depth 32, heads 16) with CPE (max 2048 px, 4 CLS + 4 register tokens) and an MLP2 adaptor head (1280 → 1536)
• Text encoder: SigLIP2-g-384 text tower (hidden 1152, 27 layers, 16 heads, projection 1536)
• Joint embedding dim: 1536
• Total parameters: 1.387 B (text tower 0.708 B)
• Native training resolution: 256 px
• Training: 15 epochs (checkpoint at epoch 14, global_step 63480), LAMB optimizer, lr 5e-7 for both towers with cosine decay, batch 32 × 8 GPUs = 256, fp16

Zero-Shot ImageNet-1K Results

Evaluated with the standard OpenAI CLIP zero-shot protocol: all 1000 ImageNet classnames, 80 OPENAI_IMAGENET_TEMPLATES prompts averaged per class then L2-normalized, short-edge bicubic resize → center crop → CLIP normalization.

The SigLIP2-g reference reproduces its published ~87% zero-shot accuracy under the same harness, confirming the evaluation protocol. The gap reflects both the lower inference resolution (256 vs 384) and the distillation loss through the RADIO adaptor head.

⚠️ Important: tokenizer caveat

transformers 5.2.0 mis-converts this SigLIP2 GemmaTokenizer — it drops the sentencepiece metaspace prefix, tokenizing "a photo of a dog" as ['ap','hoto','of','adog'] instead of ['a','▁photo','▁of','▁a','▁dog']. This silently corrupts every text embedding and collapses zero-shot accuracy to near-random.

Always tokenize with sentencepiece directly:

import sentencepiece as spm
import torch

sp = spm.SentencePieceProcessor(model_file="tokenizer.model")

def tokenize(texts, max_length=64):
    rows = []
    for t in texts:
        ids = sp.encode(t)[: max_length - 1] + [1]        # append <eos>
        rows.append(ids + [0] * (max_length - len(ids)))  # right-pad <pad>
    return torch.tensor(rows, dtype=torch.long)

Two further protocol details matter: SigLIP is trained to attend over padding, so do not apply a padding attention mask, and prompts must be canonicalized big_vision-style (text.replace('_', ' '), strip punctuation, lowercase, collapse whitespace) before encoding.

Usage

import torch, torch.nn.functional as F
from PIL import Image
from radio_clip_infer import CRADIOLite, load_tao_checkpoint, TextTokenizer, preprocess_val

model = CRADIOLite()
load_tao_checkpoint(model, "radio-clip_v1.0.pth")
model.eval()

tokenizer = TextTokenizer("siglip2_tok")
labels = ["a photo of a cat", "a photo of a dog"]
ids, mask = tokenizer(labels)

image = preprocess_val(Image.open("cat.jpg"), image_size=256).unsqueeze(0)

with torch.no_grad():
    img_f = model.encode_image(image)             # already L2-normalized
    txt_f = model.encode_text(ids, mask)          # already L2-normalized
    logits = img_f @ txt_f.T

print(logits.softmax(dim=-1))

Inference above 256 px works and can help slightly (CPE supports up to 2048 px), but 256 px matches the training resolution.

Verification

The reimplementation was validated against references before benchmarking: the text tower matches HuggingFace SigLIP2-g bit-exactly (max abs diff = 0.0), and the image path (ViTPatchGenerator + CPE + MLP2 adaptor) matches official NVIDIA RADIO bit-exactly. All 1306 tensors load with strict=True. Relative to the original C-RADIOv3-H, TAO fine-tuning drifted the backbone by 1.0e-2 and the text tower by 3.3e-3 (relative L2); logit_scale (2.3180) and logit_bias (-9.9959) were unchanged.

Limitations

• The feat_mlp head (DINOv2/SAM dense-feature alignment) received no gradient during this fine-tuning run and should not be relied upon.
• Zero-shot accuracy trails the SigLIP2-g teacher by roughly 22 points; use the teacher directly if peak zero-shot classification is the goal rather than RADIO's multi-teacher dense features.
• Evaluated on a 5000-image class-balanced ImageNet-1K validation subset, not the full 50k set; expect roughly ±0.7 pp sampling error.