Model card for naflex_ViT-B-32.cc12m-pj24-32-40
Two-tower CLIP with a NaFlexVit B/32 vision tower trained with patch-size jitter: each batch is patchified at 24, 32 or 40 px (uniform), the base 32 px projection weight resampled on the fly (FlexiViT-style pseudo-inverse resize, timm enable_patch_interpolator), on top of the usual NaFlex sequence-length sampling. Otherwise identical to naflex_ViT-B-32.cc12m, the fixed patch-32 run this card compares against. Loads with naflex_patch_interp enabled, so it accepts any patch size at inference; see the sweep below for which ones are worth using.
Research checkpoint from a NaFlex patch-size jitter experiment on CC12M (variable patch size + variable sequence length training); not a production model.
Model Details
- Dataset: pixparse/cc12m-wds
- Epochs: 32
- Precision: bf16 amp
- Weight decay: 0.25
- Objective: InfoNCE
- Vision tower: timm NaFlexVit base, base patch 32,
naflex_patch_interp=true - Patch sizes (train): 24 / 32 / 40 px, sampled uniformly per batch
- Sequence lengths (train): 36 / 49 / 64 / 81 tokens, 200704 tokens per local batch
- Text tower: classic CLIP transformer, BPE vocab 49408, ctx 77
- Optimizer: nadamuon, lr 3e-3, warmup 10k
- Batch size: 4096
Model Usage
import torch
import open_clip
model, preprocess = open_clip.create_model_from_pretrained('hf-hub:rwightman/naflex_ViT-B-32.cc12m-pj24-32-40')
tokenizer = open_clip.get_tokenizer('hf-hub:rwightman/naflex_ViT-B-32.cc12m-pj24-32-40')
image = preprocess(pil_image).unsqueeze(0)
text = tokenizer(['a dog', 'a cat', 'a bird'])
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
Model Comparison
Zero-shot ImageNet-1k top-1 (%) as a function of the eval patch size and sequence length, for this model and for naflex_ViT-B-32.cc12m (same config, recipe and data, trained at patch 32 only). Both are epoch-32 checkpoints evaluated with the same harness: OpenAI 80-prompt ensemble, the NaFlex eval transform (each image resized so its patch grid fits the token budget, so a cell's effective resolution is about sqrt(seq_len) x patch; e.g. patch 32 x 49 tokens = 224 px), and the timm patch-embed interpolator enabled for both models. Rows other than 32 for the fixed model are therefore pure post-hoc weight resampling of a patch-32-only model; the delta table isolates what the jitter training adds. Both models trained at 36-81 tokens; 100-256 is extrapolation for both.
This model (patch jitter 24 / 32 / 40)
| patch \ seq len | 36 | 49 | 64 | 81 | 100 | 144 | 196 | 256 |
|---|---|---|---|---|---|---|---|---|
| 16 | 29.05 | 30.95 | 32.33 | 33.18 | 33.72 | 33.60 | 32.83 | 31.48 |
| 24 | 38.07 | 39.63 | 40.37 | 41.06 | 41.21 | 41.17 | 40.57 | 39.65 |
| 32 | 38.29 | 39.72 | 40.39 | 41.16 | 41.40 | 41.35 | 40.66 | 39.73 |
| 40 | 38.09 | 39.45 | 40.27 | 40.98 | 41.07 | 41.12 | 40.50 | 39.64 |
| 48 | 37.97 | 39.20 | 40.16 | 40.86 | 41.05 | 41.16 | 40.53 | 39.70 |
Fixed patch 32 (naflex_ViT-B-32.cc12m)
| patch \ seq len | 36 | 49 | 64 | 81 | 100 | 144 | 196 | 256 |
|---|---|---|---|---|---|---|---|---|
| 16 | 27.15 | 29.05 | 30.19 | 31.05 | 31.48 | 31.48 | 30.66 | 29.43 |
| 24 | 36.58 | 38.25 | 39.01 | 39.80 | 40.00 | 40.22 | 40.00 | 39.01 |
| 32 | 38.51 | 39.88 | 40.92 | 41.52 | 41.68 | 41.67 | 41.41 | 40.56 |
| 40 | 36.50 | 38.26 | 39.27 | 40.24 | 40.50 | 41.09 | 41.11 | 40.30 |
| 48 | 35.84 | 37.88 | 38.90 | 39.89 | 40.37 | 41.19 | 41.06 | 40.25 |
Delta: jitter minus fixed (top-1 points)
| patch \ seq len | 36 | 49 | 64 | 81 | 100 | 144 | 196 | 256 |
|---|---|---|---|---|---|---|---|---|
| 16 | +1.90 | +1.90 | +2.14 | +2.13 | +2.24 | +2.12 | +2.17 | +2.05 |
| 24 | +1.49 | +1.38 | +1.36 | +1.26 | +1.21 | +0.95 | +0.57 | +0.64 |
| 32 | -0.22 | -0.16 | -0.53 | -0.36 | -0.28 | -0.32 | -0.75 | -0.83 |
| 40 | +1.59 | +1.19 | +1.00 | +0.74 | +0.57 | +0.03 | -0.61 | -0.66 |
| 48 | +2.13 | +1.32 | +1.26 | +0.97 | +0.68 | -0.03 | -0.53 | -0.55 |
Reads:
- Patch size barely matters between 24 and 48 for this model: at any sequence length the four rows sit within ~0.3 pt of each other, including 48 px, which it never trained on. Accuracy is set by the token count. The fixed model needs 144+ tokens before its off-base rows come within a point of base.
- Jitter is ~free at the base patch (-0.2 to -0.5 pt vs fixed at patch 32, inside the <=0.6 pt repeat-run noise of these runs) and buys 1-2 pt in the cheap regime: at 36-81 tokens every off-base patch is 1.0-2.1 pt better than post-hoc resampling alone (e.g. patch 48 x 36 tokens, a 288 px image in 36 tokens: 38.0 vs 35.8).
- Post-hoc resampling already covers the high-token regime: at 144+ tokens the fixed model is within a point of its base at every patch >= 24, and at 196-256 tokens it is ~0.5 pt ahead of the jittered model, which decays slightly faster past the trained sequence lengths.
- Patch 16 does not work for either model (~30 vs ~41): downsampling the 32 px kernel loses information that upsampling to 48 does not, and 16 was outside the jitter range.
- Sequence length extrapolates for both: both peak at 100-144 tokens against a 36-81 training range (41.4 / 41.7 top-1 at patch 32 x 100 tokens), falling off gently by 256.
The broader benchmark table for the CC12M B/32 comparison runs (ImageNet variants, COCO retrieval, captioning) is on the naflex_ViT-B-32.cc12m card; this checkpoint was not run through that harness.
To evaluate or train at another patch size with the open_clip trainer, pass e.g.
--use-naflex --naflex-patch-sizes 24 --naflex-seq-lens 100 (the interpolator is already enabled by
this model's config; for other NaFlex checkpoints add --force-naflex-patch-interp).