Model card for mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3
MaMMUT2 with a modern text decoder (RoPE, RMSNorm, SwiGLU, gated attention, qk-norm) and tiktoken r50k vocab at ctx 128. Best moderntext run in the comparison.
Research checkpoint from a B/32-scale architecture/objective comparison on CC12M (CLIP vs MaMMUT2 vs modern text decoder); not a production model.
Model Details
- Dataset: pixparse/cc12m-wds
- Epochs: 32
- Precision: bf16 amp
- Weight decay: 0.25
- Objective: InfoNCE + 2x caption CE + 1e-5 z-loss
- Text decoder: modern (RoPE/RMSNorm/SwiGLU/gated attn), qk-norm, tiktoken r50k vocab 50260, ctx 128, variable-length text
- Optimizer: adamw, lr 2e-3
- Batch size: 1024
Model Usage
import torch
import open_clip
model, preprocess = open_clip.create_model_from_pretrained('hf-hub:rwightman/mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3')
tokenizer = open_clip.get_tokenizer('hf-hub:rwightman/mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3')
image = preprocess(pil_image).unsqueeze(0)
text = tokenizer(['a dog', 'a cat', 'a bird'])
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
Caption generation:
with torch.no_grad():
tokens = model.generate(image, generation_type='beam_search', seq_len=30)
print(tokenizer.decode(tokens[0].cpu().tolist()))
Model Comparison
| run | IN1K | IN-V2 | IN-R | IN-Sk | I2T R@1 | T2I R@1 | CIDEr | BLEU-4 | METEOR |
|---|---|---|---|---|---|---|---|---|---|
| clip-siglip | 42.72 | 36.44 | 54.38 | 32.08 | 34.84 | 25.95 | |||
| clip-naflex | 40.4 | 34.43 | 50.53 | 28.55 | 34.6 | 24.25 | |||
| mammut2 (no z) | 37.8 | 31.98 | 48.8 | 27.16 | 37.52 | 24.72 | 10.02 | 2.43 | 7.37 |
| mammut2 (+z) | 38.36 | 32.47 | 49.44 | 27.12 | 38.18 | 24.54 | 10.04 | 2.41 | 7.38 |
| mammut2-naflex | 37.54 | 31.92 | 48.63 | 26.28 | 38.64 | 24.7 | 9.52 | 2.33 | 7.4 |
| mammut2-mt no-qk lr1e-3 | 38.07 | 32.78 | 50.84 | 27.28 | 37.44 | 25.18 | 10.96 | 3.58 | 7.69 |
| mammut2-mt qk lr2e-3 (this model) | 38.37 | 33.02 | 50.56 | 27.18 | 37.94 | 24.93 | 10.85 | 3.44 | 7.57 |
| mammut2-mt qk lr2e-3 b4096 | 40.0 | 34.21 | 53.91 | 29.81 | 37.7 | 25.37 | 12.23 | 3.91 | 7.99 |
| mammut2-mt qk lr3e-3 | 37.05 | 31.82 | 48.41 | 26.03 | 35.92 | 24.06 | 10.29 | 3.31 | 7.35 |
All eight runs: CC12M, 32 epochs, 2x GPU, bf16 amp, wd 0.25, evaluated at epoch 32 with one harness (ImageNet zero-shot w/ OpenAI 80-prompt ensemble; COCO Karpathy-test retrieval, I2T = any-of-5-captions; caption metrics vs the 5 COCO refs, beam search). Key reads:
- The CLIP rows are prior runs, not purpose-trained controls: they were the closest available baselines but come from NAdaMuon optimizer experiments (nadamuon @ bs 4096 vs the MaMMUT runs' adamw @ bs ~1024), so cross-family deltas bundle optimizer + batch + loss.
- Caption objective -> I2T retrieval: every MaMMUT2 run beats both CLIP runs by 1-4 pt I2T R@1 despite 1/4 the contrastive batch and a plainer optimizer.
- Zero-shot ordering tracks batch/loss/optimizer, not architecture -- confirmed by the bs-4096 moderntext run: at matched batch (still adamw) it lands within 0.4 IN1K of clip-naflex and within 0.5 IN-R of clip-siglip. Batch was most of the CLIP lead.
- The caption-objective I2T advantage is batch-independent (37.7-38.6 I2T R@1 at both bs 1024 and 4096), while zero-shot, T2I, and caption metrics all scale with batch.
- NaFlex vision and 1e-5 caption z-loss are both ~free (within the <=0.6 pt repeat-run noise).
- Modern text tower (bundled with r50k tokenizer + ctx 128) at its best LR: IN1K parity with classic, +V2/+IN-R/+T2I, and clearly better caption metrics. qk-norm buys 2x LR headroom rather than direct quality; 3e-3 is past the stable edge even with it.
- Caption scores are style-limited: CC12M alt-text register vs COCO references. Absolute CIDEr is low for all runs; relative differences are the signal.