Model Details
MoE-ViE is a family of Mixture-of-Experts vision encoders for image and video understanding, trained with a contrastive vision-language recipe. Each MoE block keeps a shared always-on expert alongside a routed pool, so only a small fraction of the weights is used for any given token.
Model Developer: Meta
This checkpoint: MoEViE-H14-448 — H/14 at 448px.
Model Configurations
MoE-ViE comes in three sizes. Only active experts run per token, so the activated
parameter count is far below the total.
| Scale | Width | Depth | Experts (active/total) | Total params | Activated params | Resolution |
|---|---|---|---|---|---|---|
| B/16 | 768 | 12 | 4 / 32 | 0.5B | 0.1B | 224px |
| L/16 | 1024 | 24 | 4 / 32 | 1.7B | 0.3B | 384px |
| H/14 | 1280 | 32 | 8 / 32 | 3.5B | 1.1B | 448px |
Every MoE block has 1 shared expert that is always active plus a routed pool; the
active/total column counts the shared expert. Routing is per-token top-k on a sigmoid gate.
Model Performance
Zero-shot results. Top-1 accuracy (%) for classification, recall@1 (%) for retrieval.
| Model | Checkpoint | IN-1k | ObjectNet | COCO-T2I | Kinetics-400 | MSR-VTT-T2V |
|---|---|---|---|---|---|---|
| B/16 224px | MoEViE-B16-224 | 79.3 | 74.4 | 52.1 | 68.3 | 47.9 |
| L/16 384px | MoEViE-L16-384 | 83.6 | 85.0 | 57.2 | 74.5 | 50.5 |
| H/14 448px | MoEViE-H14-448 | 85.1 | 87.0 | 56.8 | 76.9 | 51.6 |
How to use
Install
git clone https://github.com/facebookresearch/moe_vie
cd moe_vie
pip install -r requirements.txt
Requires a CUDA GPU — the Mixture-of-Experts kernels are compiled with Triton at runtime.
Image and text feature extraction
import torch
from PIL import Image
from open_clip import create_model_and_transforms, get_tokenizer, image_to_device
MEAN, STD = (0.5, 0.5, 0.5), (0.5, 0.5, 0.5)
model, _, preprocess = create_model_and_transforms(
"MoEViE-H14-448",
pretrained=True, # downloads from the Hub
force_preprocess_cfg=dict(
patch_size=14, size_range=(448, 448), center_crop=True, window_size=1
),
image_mean=MEAN, image_std=STD,
)
model = model.cuda().eval()
tokenizer = get_tokenizer("MoEViE-H14-448")
labels = ["a diagram", "a dog", "a cat"]
packed, _ = preprocess.collate_fn([(preprocess(Image.open("cat.png").convert("RGB")), 0)])
packed = image_to_device(packed, "cuda", torch.float32, mean=MEAN, std=STD)
text = tokenizer(labels).cuda()
with torch.no_grad(), torch.autocast("cuda"):
image_features = model.encode_image(packed, normalize=True)
text_features = model.encode_text(text, normalize=True)
probs = (model.logit_scale.exp() * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", probs)
See demo/demo.py and the repository README for the
zero-shot evaluation suite.
License
Released under CC BY-NC 4.0 — non-commercial research use.
Citation
If you find this work useful, please cite:
@article{zhang2026moevie,
title={MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding},
author={Bonan Zhang and Shiyu Dong and Quan Hung Tran and Katharina Gschwind and Shuqi Yang and Sijia Chen and Adel Ahmadyan and Seungwhan Moon and Lu Zhang and Ahmed Kirmani and Babak Damavandi and Anuj Kumar},
journal={arXiv preprint arXiv:2608.17402},
year={2026}
}