Model Card for Pixio
Pixio is a family of versatile self-supervised vision foundation models. Pixio produces competitive dense features by simple masked autoencoding (MAE) on 2B web-crawled images with minimal human curation.
Pixio enhances MAE pre-training framework by using a deeper decoder, masking at a larger granularity, and introducing additional class tokens.
Model Details
As described in the Pixio paper, 5 models are provided:
- 1 ViT-5B trained from scratch,
- 4 ViT-B/L/H/1B models distilled from the ViT-5B
Each model takes an image as input and returns eight class tokens and patch tokens. These models follow a standard ViT architecture, with a patch size of 16. For a 256x256 image, this results in 8 class tokens + 256 patch tokens = 264 tokens.
The models can accept larger images provided the image shapes are multiples of the patch size (16).
Model Description
- Developed by: FAIR at Meta, HKU
- Model type: Vision Transformer
- License: FAIR Noncommercial Research License
Model Sources
- Repository: https://github.com/facebookresearch/pixio
- Paper: In Pursuit of Pixel Supervision for Visual Pre-training
How to use
Here is how to use this model:
from transformers import AutoImageProcessor, AutoModel
from PIL import Image
import requests
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
processor = AutoImageProcessor.from_pretrained('facebook/pixio-vit5b16')
model = AutoModel.from_pretrained('facebook/pixio-vit5b16')
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs, output_hidden_states=True)
last_hidden_states_norm = outputs.last_hidden_state # 8 class tokens + patch tokens after last LayerNorm
last_hidden_states = outputs.hidden_states[-1] # 8 class tokens + patch tokens before last LayerNorm
Citation
@article{pixio,
title={In Pursuit of Pixel Supervision for Visual Pre-training},
author={Yang, Lihe and Li, Shang-Wen and Li, Yang and Lei, Xinjie and Wang, Dong and Mohamed, Abdelrahman and Zhao, Hengshuang and Xu, Hu},
journal={arXiv:2512.15715},
year={2025}
}