Slot-RAE (DINOv3-B, COCO)
Checkpoint of Slot-RAE: Streamlining Object-Centric Learning via Direct Representation Auto-Encoders (under review).
Slot-RAE learns 7 object slots (768-d, plus 1 global token) on top of a frozen DINOv3 ViT-B/16 encoder and trains a slot-conditioned Diffusion Transformer (Base, with a wide DDT head and a REPA head) from scratch to regenerate the DINOv3 features. A frozen RAEv2 decoder maps generated features to pixels. This enables zero-shot object removal, insertion and swapping.
- 🌐 Project page: https://slot-rae.github.io/
- 💻 Code: https://github.com/alexcbb/slot-rae
Files
| File | Content |
|---|---|
slot_rae_dinov3_b_coco.ckpt | Slot encoder + DiT weights (1.2 GB). The frozen DINOv3 encoder and RAE decoder are loaded from nyu-visionx/RAEv2-models. |
Usage
from huggingface_hub import hf_hub_download
from slot_rae import load_slot_rae # pip install -e . from the code repository
ckpt = hf_hub_download("Beegbrain/slot-rae-dinov3-b-coco", "slot_rae_dinov3_b_coco.ckpt")
model = load_slot_rae(ckpt, device="cuda") # RAE weights: see the code README (RAE_ROOT)
out = model(images) # ImageNet-normalized (B, 3, 256, 256)
out["slots"], out["masks"], out["recon_image"]
Results (COCO val2017)
| Object discovery | mBOi | mBOc | mIoUi |
|---|---|---|---|
| Paper (3 seeds) | 32.3 | 38.1 | 30.8 |
| This checkpoint (3 seeds) | 32.5 | 39.1 | 31.0 |
License
CC BY-NC 4.0, inherited from RAEv2.