Helium1-VL-2B
Helium1-VL-2B is an instruct-tuned vision-language model (VLM) based on the Helium1-2B text-only language model and a pretrained vision encoder from Qwen-2.5VL.
This model is released as part of the CASA project. While the CASA architecture focuses on cross-attention fusion, Helium1-VL-2B serves as a high-performance token insertion baseline, achieving state-of-the-art results among models of comparable size trained on publicly available datasets.
- Paper: CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
- Project Page: https://kyutai.org/casa
- GitHub Repository: https://github.com/kyutai-labs/casa
Sample Usage
You can run inference using the following code snippet. This model requires trust_remote_code=True to load the custom architecture.
import torch
from transformers.models.auto.modeling_auto import AutoModel
from transformers.models.auto.processing_auto import AutoProcessor
model_id = "kyutai/Helium1-VL-2B"
model = AutoModel.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
trust_remote_code=True,
).cuda()
processor = AutoProcessor.from_pretrained(
model_id,
trust_remote_code=True,
)
conversation = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.png",
},
{
"type": "text",
"text": "Describe this image.",
},
],
},
]
inputs = processor.tokenize_messages(messages=conversation)
inputs = inputs.to(model.device)
input_len = inputs["input_ids"].shape[1]
output_ids = model.generate_from_image(
**inputs,
max_new_tokens=512,
pre_image_tokens=processor.pre_image_tokens,
post_image_tokens=processor.post_image_tokens,
eos_token_id=model.generation_config.eos_token_id,
)[0, input_len:]
response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)
Citation
If you use this model or the CASA fusion paradigm in your research, please cite:
@article{kyutai2025casa,
author = {Moritz B\"ohle and Am\'elie Royer and Juliette Marrie and Edouard Grave and Patrick P\'erez},
year = {2025},
title = {CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion},
journal = {ArXiv},
url = {https://arxiv.org/abs/2512.19535}
}