Sa2VA-LLaVA-1.5-7B
Sa2VA-LLaVA-1.5-7B is a Sa2VA model built on LLaVA-1.5-7B
(CLIP-ViT-L-336 vision encoder + Vicuna-7B language model) with a SAM2
grounding encoder. The MLLM predicts a [SEG] token whose hidden state
conditions the SAM2 mask decoder, producing dense image and video referring
segmentation alongside open-ended chat. It is intended as a LISA-comparable
baseline within the Sa2VA family.
This checkpoint is self-contained: the SAM2 grounding code is vendored into
the repository, so it loads with trust_remote_code=True without any extra
packages.
Results
Image referring segmentation (cIoU):
| RefCOCO val / testA / testB | RefCOCO+ val / testA / testB | RefCOCOg val / test |
|---|---|---|
| 80.3 / 82.4 / 76.7 | 73.1 / 78.0 / 66.2 | 79.2 / 80.1 |
Video referring segmentation (J&F):
| MeViS (val_u) | ReVOS | Ref-DAVIS17 |
|---|---|---|
| 54.8 | 54.0 | 74.6 |
Grounded conversation generation (GCG, val):
| AP50 | mIoU | Recall |
|---|---|---|
| 31.2 | 66.6 | 43.7 |
Usage
import torch
from transformers import AutoModel, AutoTokenizer
from PIL import Image
path = "HarborYuan/Sa2VA-LLaVA-1.5-7B"
model = AutoModel.from_pretrained(
path, torch_dtype=torch.bfloat16, trust_remote_code=True, low_cpu_mem_usage=True,
).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
image = Image.open("your_image.jpg").convert("RGB")
out = model.predict_forward(
image=image,
text="<image>Please segment the dog in the image.",
tokenizer=tokenizer,
)
print(out["prediction"]) # text response containing [SEG]
masks = out["prediction_masks"] # list of boolean masks at the original image size
Load in bfloat16 (the lm_head is kept in higher precision; torch_dtype="auto"
mixes dtypes and fails). This model uses a tokenizer (not an AutoProcessor).
For video, pass video=[frame0, frame1, ...] (a list of PIL images) instead of image.
Notes
- SAM2 grounding input resolution is 1024.
- Built on Sa2VA.
Citation
If you find this project useful in your research, please consider citing:
@article{sa2va,
title={Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos},
author={Yuan, Haobo and Li, Xiangtai and Zhang, Tao and Sun, Yueyi and Huang, Zilong and Xu, Shilin and Ji, Shunping and Tong, Yunhai and Qi, Lu and Feng, Jiashi and Yang, Ming-Hsuan},
journal={IEEE TPAMI},
year={2026}
}