Model Card for rope_vit_reg8_b14_nps_avg_capi-dino-bio
A RoPE ViT Reg8 B/14 image encoder with average pooling, pretrained using CAPI-DINO on natural biological images. This model has not been fine-tuned for a specific classification task and is intended to be used as a general-purpose feature extractor or a backbone for downstream tasks like object detection, segmentation, or custom classification.
Model Details
-
Model Type: Image encoder and detection backbone
-
Model Stats:
- Params (M): 85.7
- Input image size: 224 x 224
-
Dataset: Trained on a diverse dataset of approximately 31M images, including:
- TreeOfLife-10M-EOL-NaturalImages
- iNaturalist 2021
- BIOSCAN-5M (pretrain split)
- TreeOfLife-200M (subset)
- IP102 v1.1
- iWildCam 2022 (subset)
- The Birder dataset (private dataset)
-
Papers:
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: https://arxiv.org/abs/2010.11929
- Rotary Position Embedding for Vision Transformer: https://arxiv.org/abs/2403.13298
- Vision Transformers Need Registers: https://arxiv.org/abs/2309.16588
- Cluster and Predict Latent Patches for Improved Masked Image Modeling: https://arxiv.org/abs/2502.08769
Model Usage
Image Embeddings
import birder
from birder.inference.classification import infer_image
# Option 1: manual setup (more control over preprocessing)
net, model_info = birder.load_pretrained_model("rope_vit_reg8_b14_nps_avg_capi-dino-bio", inference=True)
# Get the image size the model was trained on
size = birder.get_size_from_signature(model_info.signature)
# Create an inference transform
transform = birder.classification_transform(size, model_info.rgb_stats)
# Option 2: helper (quick start with default preprocessing)
net, model_info, transform = birder.load_pretrained_model_and_transform("rope_vit_reg8_b14_nps_avg_capi-dino-bio", inference=True)
image = "path/to/image.jpeg" # or a PIL image
out, embedding = infer_image(net, image, transform, return_embedding=True)
# embedding is a NumPy array with shape of (1, 768)
Detection Feature Map
from PIL import Image
import birder
net, model_info, transform = birder.load_pretrained_model_and_transform("rope_vit_reg8_b14_nps_avg_capi-dino-bio", inference=True)
image = Image.open("path/to/image.jpeg")
features = net.detection_features(transform(image).unsqueeze(0))
# features is a dict (stage name -> torch.Tensor)
print([(k, v.size()) for k, v in features.items()])
# Output example:
# [('stage1', torch.Size([1, 768, 16, 16]))]
Citation
@misc{dosovitskiy2021imageworth16x16words,
title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
author={Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
year={2021},
eprint={2010.11929},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2010.11929},
}
@misc{heo2024rotarypositionembeddingvision,
title={Rotary Position Embedding for Vision Transformer},
author={Byeongho Heo and Song Park and Dongyoon Han and Sangdoo Yun},
year={2024},
eprint={2403.13298},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2403.13298},
}
@misc{darcet2024visiontransformersneedregisters,
title={Vision Transformers Need Registers},
author={Timothée Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
year={2024},
eprint={2309.16588},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2309.16588},
}
@misc{darcet2025clusterpredictlatentpatches,
title={Cluster and Predict Latent Patches for Improved Masked Image Modeling},
author={Timothée Darcet and Federico Baldassarre and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
year={2025},
eprint={2502.08769},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2502.08769},
}