Model card for naflex_ViT-B-32.cc12m-pj24-32-40-siglip

Two-tower CLIP with a NaFlexVit B/32 vision tower, trained on CC12M with the SigLIP sigmoid pairwise loss and patch-size jitter at 24 / 32 / 40 px. The text tower uses classic CLIP BPE (49408 tokens, context length 77). The exported config enables patch-embedding interpolation and includes the learned logit bias. Best measured ImageNet-1k zero-shot top-1 is 43.778% at patch 32 / 100 image tokens; see the full comparison with the InfoNCE patch-jitter run below.

Research checkpoint from a NaFlex patch-size jitter experiment on CC12M. The comparison below covers these two trained runs; it does not estimate variation across seeds.

Model Details

• Dataset: pixparse/cc12m-wds
• Epochs: 32
• Precision: bf16 amp
• Weight decay: 0.25
• Objective: SigLIP (sigmoid pairwise, learned bias)
• Loss implementation: --siglip --siglip-chunk-size 1024 --loss-dist-impl gather
• Vision tower: timm NaFlexVit base, base patch 32, naflex_patch_interp=true
• Patch sizes (train): 24 / 32 / 40 px, sampled uniformly per batch
• Sequence lengths (train): 36 / 49 / 64 / 81 tokens, sampled uniformly per batch
• Image token budget (per rank): 200704 per batch; image batch size varies with sequence length
• Reference batch size (per rank): 4096
• Text tower: classic CLIP transformer, BPE vocab 49408, ctx 77
• Optimizer: timm/nadamuon, lr 3e-3, warmup 2000 steps
• Augmentation: NaFlex + timm, random crop scale (0.4, 1.0), random erasing probability 0.3
• Training samples per epoch: 10968539
• GPUs: 2
• Seed: 0
• Checkpoint: epoch_32.pt from naflex_ViT-B-32_cc12m_pj24-32-40-sig2

Model Usage

NaFlex inference at patch 32 / 100 tokens, the best setting in the grid below. This example uses the same evaluation transform as the sweep and requires timm with NaFlex support (evaluated with timm 1.0.28 and PyTorch 2.9.1).

import torch
import open_clip
from PIL import Image

model_id = 'hf-hub:rwightman/naflex_ViT-B-32.cc12m-pj24-32-40-siglip'
model, _, eval_transform = open_clip.create_model_and_transforms(
    model_id, aug_cfg={'use_timm': True, 'naflex': True},
)
model.eval()
tokenizer = open_clip.get_tokenizer(model_id)

patch_size, token_budget = 32, 100
preprocess = eval_transform(
    max_seq_len=token_budget,
    patch_size=(patch_size, patch_size),
    flatten_patches=patch_size == 32,  # preserve spatial dimensions for non-base patches
)
sample = preprocess(Image.open('image.jpg').convert('RGB'))
image = {key: value.unsqueeze(0) for key, value in sample.items()}
text = tokenizer(['a dog', 'a cat', 'a bird'])
with torch.inference_mode():
    image_features = model.encode_image(image, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    logits = model.logit_scale.exp() * image_features @ text_features.T + model.logit_bias
    scores = logits.sigmoid()

Change patch_size and token_budget to select another grid cell. The example processes a single image; batches of different aspect ratios need NaFlex collation and validity masks. The sigmoid scores are independent image-text match scores. The zero-shot grid uses the 80-prompt class ensemble.

Model Comparison

Zero-shot ImageNet-1k top-1 (%) over evaluation patch size x image token budget, comparing this SigLIP checkpoint with the InfoNCE patch-jitter checkpoint. Both are epoch-32 CC12M runs trained with patch sizes 24 / 32 / 40 and sequence lengths 36 / 49 / 64 / 81, using NaDAMuon at lr 3e-3, 2000 warmup steps, and 200704 image tokens per local batch. Token budgets 100-256 and patch sizes 16 / 48 extend beyond the training choices.

Both grids were evaluated together with scripts/naflex_imagenet_sweep.py (revision 9782671): all 50000 ImageNet-1k validation images, the OpenAI 80-prompt ensemble, BF16 autocast, batch size 256, the trainer's NaFlex evaluation transform, and patch-embedding interpolation enabled for both models. Images retain their aspect ratio and are resized to fit the patch budget. For a square image, the approximate resolution is patch_size * sqrt(token_budget); patch 32 / 100 tokens is about 320 px.

This model: SigLIP top-1 (%)

InfoNCE patch jitter: rerun top-1 (%)

SigLIP minus InfoNCE (percentage points)

• Both peak at patch 32 / 100 tokens: SigLIP 43.778% top-1 / 70.710% top-5, versus InfoNCE 41.400% / 68.654%, a 2.378-point top-1 improvement.
• SigLIP improves all 40 / 40 settings: mean +2.234 points, range +1.318 to +3.102.
• At 144 tokens, SigLIP's patch 24-48 results span just 0.086 points (43.588-43.674%). Patch 16 is substantially weaker, peaking at 35.340%; both models lose accuracy beyond 144 tokens.
• The InfoNCE rerun reproduces its previously published grid within 0.090 points in every cell (mean absolute difference 0.022 points). The precise source of the small remaining numerical differences has not been isolated.

This checkpoint has only been evaluated on ImageNet-1k so far; no ImageNet robustness, COCO retrieval, or captioning results are reported here. These are comparisons of two runs, without a multi-seed estimate of training variance.

For the open_clip trainer, use --use-naflex --naflex-patch-sizes 32 --naflex-seq-lens 100 for the best measured setting. When loading the original training checkpoint by its base model name, also pass --siglip --force-naflex-patch-interp; the Hub config already preserves the bias and interpolator.