ConCor-1

Vision-Language Grounding as Bidirectional Concept Correspondence

Jieyu Zhang1*, Ziqi Gao1,2*, Luke Zettlemoyer1,3, Ranjay Krishna1

1University of Washington · 2Allen Institute for AI · 3FAIR at Meta
*Equal contribution

Quick links

📄 Paper · 🌐 Project page · 💻 GitHub · 🤗 Data · 🎨 Demo

Model Summary

ConCor-1 is a vision-language grounding model designed to explicitly predict bidirectional concept correspondences between language and visual content. Built on a pretrained Qwen3.5-0.8B vision-language backbone, it introduces a set of learnable bridge tokens, each representing a candidate text–image correspondence. By jointly attending to visual and textual tokens, these bridge tokens aggregate multimodal context to represent potential alignments between image regions and text spans. For each bridge token, ConCor-1 simultaneously predicts a text mask identifying the grounded text segment, an image mask localizing the corresponding visual instance, and a presence score indicating whether the proposed pairing is a valid correspondence.

Usage

Environment

The released checkpoint is supported with the following core inference stack:

Install a PyTorch/torchvision build compatible with your CUDA environment first. Then install the remaining inference dependencies:

pip install "transformers==5.3.0" "safetensors>=0.4.0" \
  "huggingface_hub>=0.30.0" "numpy>=1.24" "pillow>=10.0" \
  "flash-linear-attention>=0.4.1"

The example below uses PyTorch SDPA for the backbone's full-attention layers. To use FlashAttention-2, install flash-attn>=2.8.0 and set attn_implementation="flash_attention_2". causal-conv1d>=1.4.0 is an optional acceleration dependency for the linear-attention layers.

Image + caption

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_id = "UWGZQ/ConCor-1"  # or a local path to this repository

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",  # "flash_attention_2" also works
).to("cuda").eval()

image = Image.open("example.png").convert("RGB")
text = (
    "This image depicts a close-up of a brown bear in a natural outdoor setting. "
    "The background consists of lush green grass. In the foreground, a large brown bear "
    "is positioned centrally."
)
# A referring expression works the same way:
# text = "the large brown bear in the foreground"

inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    outputs = model(**inputs)

correspondences = processor.post_process_correspondences(
    outputs, text=text, target_sizes=[(image.height, image.width)]
)[0]

for correspondence in correspondences:
    print(
        f"{correspondence['presence_score']:.3f}",
        correspondence["text_phrases"],   # phrases of this correspondence's text mask
        correspondence["text_spans"],     # character spans into `text`
        correspondence["mask"].shape,     # bool array, (height, width)
    )

post_process_correspondences also takes presence_threshold, text_threshold, image_threshold and nms_iou_threshold (defaults 0.1, 0.45, 0.45, 0.5).

example_inference.py wraps this up as a script, including a mask-overlay renderer:

python example_inference.py \
    --image example.png \
    --text "This image depicts a close-up of a brown bear in a natural outdoor setting. The background consists of lush green grass. In the foreground, a large brown bear is positioned centrally." \
    --output overlay.png

Image + category list

image = Image.open("example_2.png").convert("RGB")
text = "baseball player . baseball glove . grass . fence . dog"

inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    outputs = model(**inputs)

for correspondence in processor.post_process_correspondences(
    outputs, text=text, target_sizes=[(image.height, image.width)]
)[0]:
    print(f"{correspondence['presence_score']:.3f}", correspondence["text_phrases"])

Batched image–text pairs

images = [Image.open("example.png").convert("RGB"), Image.open("example_2.png").convert("RGB")]
texts = [
    "This image depicts a close-up of a brown bear in a natural outdoor setting. "
    "The background consists of lush green grass. In the foreground, a large brown bear "
    "is positioned centrally.",
    "baseball player . baseball glove . grass . fence . dog",
]

inputs = processor(images=images, text=texts, return_tensors="pt").to("cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    outputs = model(**inputs)

results = processor.post_process_correspondences(
    outputs,
    text=texts,                                                        # the same list of texts
    target_sizes=[(image.height, image.width) for image in images],    # one (height, width) per sample
)

for index, correspondences in enumerate(results):
    print(f"sample {index}: {len(correspondences)} correspondence(s)")
    for correspondence in correspondences:
        print(f"  {correspondence['presence_score']:.3f}", correspondence["text_phrases"])

Files

Demo

An interactive ZeroGPU demo lives at UWGZQ/ConCor-1-demo.

License and Use

The weights and the code in this repository are released under the Apache 2.0 license. The backbone is Qwen3.5-0.8B, also Apache 2.0.

Citation

@article{zhang2026concor,
  title   = {Vision-Language Grounding as Bidirectional Concept Correspondence},
  author  = {Zhang, Jieyu and Gao, Ziqi and Zettlemoyer, Luke and Krishna, Ranjay},
  year    = {2026}
}