EmObserver

Official model release for “MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement.”

EmObserver is an emotion-oriented multimodal large language model built from Qwen3-VL-8B-Thinking. It is optimized through a four-stage training recipe for visual emotion understanding, Emotion Statement Judgement (ESJ), emotion classification, and single-/multi-image affective reasoning.

The model accepts one or more images with a natural-language question and produces a reasoning-style response followed by a concise answer:

<think>...</think><answer>...</answer>

🔗 Project map

🚀 Quick start with Transformers

Install a compatible PyTorch build first, then install the model dependencies:

python3 -m pip install -U "transformers>=5.9.0" accelerate pillow

The following example uses the same MVEI question form as the released test metadata:

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "wudq/EmObserver"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

image_url = (
    "https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/"
    "benchmarks/MVEI/images/contentment/contentment_14236.jpg"
)
prompt = (
    "Is the following statement correct about the image? Upon viewing this image, "
    "observers, despite various individual or contextual factors, are most likely "
    "to experience negative emotions. Choose the answer from "
    "{'A': 'Correct.', 'B': 'Incorrect.'}. Answer in the format of "
    "<think>...</think><answer>...</answer>."
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": image_url},
            {"type": "text", "text": prompt},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated = model.generate(**inputs, max_new_tokens=1024)
trimmed = [output[len(input_ids):] for input_ids, output in zip(inputs.input_ids, generated)]
response = processor.batch_decode(
    trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(response)

For multi-image questions, add each image to content before the text item:

"content": [
    {"type": "image", "url": first_image_url},
    {"type": "image", "url": second_image_url},
    {"type": "text", "text": pair_prompt},
]

FlashAttention 2 can reduce memory use on supported hardware. Install a build compatible with your CUDA/PyTorch stack, then pass attn_implementation="flash_attention_2" to from_pretrained.

🧪 Full evaluation with the official code

The official repository provides the vLLM inference wrapper, benchmark loaders, output parsing, and metric calculators:

git clone https://github.com/wdqqdw/EmObserver.git
cd EmObserver

python3 -m pip install -r requirements.txt
hf download wudq/MVEI_PLUS --repo-type dataset --local-dir public_data
hf download wudq/EmObserver --local-dir models/EmObserver

python3 -m evaluate.infer_and_eval \
  --engine qwen3_vl_vllm \
  --size emobserver \
  --gpu_n 8

Adjust --gpu_n to the visible GPUs that can hold the checkpoint. Use --bench to select a subset from MVEI, VECBench, EEmo-Bench-Single-Perception, and EEmo-Bench-Pair-Perception.

💬 Benchmark-style examples

The examples below reproduce question forms from the standardized test metadata. The listed answer is the reference final answer; reasoning text may vary.

1. Emotion Statement Judgement — MVEI

Image: contentment_14236.jpg

Is the following statement correct about the image? Upon viewing this image,
observers, despite various individual or contextual factors, are most likely
to experience negative emotions. Choose the answer from
{'A': 'Correct.', 'B': 'Incorrect.'}. Answer in the format of
<think>...</think><answer>...</answer>.

Reference final answer:

<answer>Incorrect.</answer>

2. Single-image emotion perception — EEmo-Bench

Image: EEmo-Bench single image 1

What do you think of surprise as one of the three main emotions you felt from
this image? Choose the answer from {'A': 'No', 'B': 'Yes'}. Answer in the
format of <think>...</think><answer>...</answer>.

Reference final answer:

<answer>Yes.</answer>

3. Image-pair emotion comparison — EEmo-Bench

Images: image 579 and image 100

Which emotion is evoked in both images and is common to them? Choose the
answer from {'A': 'Neutral', 'B': 'Fear', 'C': 'Sadness', 'D': 'Joy'}.
Answer in the format of <think>...</think><answer>...</answer>.

Reference final answer:

<answer>Joy.</answer>

4. Visual emotion classification — VECBench

Image: abstract_0001.jpg

Which emotion might this image evoke? Choose the most likely one from
['Amusement', 'Anger', 'Awe', 'Content', 'Disgust', 'Excitement', 'Fear',
'Sad']. Think step by step. Respond in the format:
<think>{your reasoning}</think><answer>{your final answer}</answer>.

Reference final answer:

<answer>Content.</answer>

📊 Released evaluation results

These values are taken from the released EmObserver inference outputs in MVEI_PLUS. They are intended to make the checkpoint release traceable; environment or decoding changes may produce different results.

🏋️ Training overview

EmObserver is produced by the four-stage public training pipeline:

1. supervised fine-tuning;
2. GRPO-based reinforcement learning;
3. on-policy self-distillation (OPSD);
4. GRPO with an LLM-consistency reward.

The runnable scripts are in the official code repository, and their released inputs are under MVEI_PLUS/training_data. External judge credentials are not included; users must provide their own endpoint when reproducing the final stage.

⚠️ Intended use and limitations

EmObserver is intended for research on visual emotion understanding, affective image analysis, multimodal reasoning, and benchmark development.

• Visual emotion is inherently subjective and culturally/contextually dependent. A model prediction should not be treated as a universal description of how every person will feel.
• The model may inherit social, cultural, demographic, and content biases from its base model and training data.
• The checkpoint is not designed for mental-health diagnosis, psychological profiling, surveillance, hiring, education assessment, or other high-stakes decisions about individuals.
• Generated reasoning can be plausible but factually or emotionally incorrect. Verify outputs when reliability matters.
• Users are responsible for reviewing the licenses, attribution requirements, privacy constraints, and redistribution terms of input images and downstream datasets.

📄 License

The checkpoint is released under the Apache 2.0 license, following its Qwen3-VL-8B-Thinking base model. Third-party datasets and images retain their respective licenses and terms.

📝 Citation

If you use EmObserver or the expanded training and evaluation release, please cite the EmObserver paper below. When using the MVEI benchmark, dataset, or Emotion Statement Judgement formulation, please also cite the original ICLR 2026 paper.

@article{wu2026mvei_emobserver,
  title   = {MVEI \& EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement},
  author  = {Wu, Daiqing and Yang, Dongbao and Yao, Jiashu and Zhang, Hongrui and Ma, Can and Zhou, Yu and Zhao, Sicheng},
  journal = {arXiv preprint arXiv:2607.21061},
  year    = {2026}
}

@inproceedings{wu2026customizing,
  title     = {Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach},
  author    = {Wu, Daiqing and Yang, Dongbao and Zhao, Sicheng and Ma, Can and Zhou, Yu},
  booktitle = {International Conference on Learning Representations},
  year      = {2026},
  url       = {https://openreview.net/forum?id=dQTSXWqZws}
}