HumanSense_Omni_Reasoning

Hugging Face Paper                arXiv:2508.10576                Homepage                GitHub                Hugging Face (data)                Hugging Face (model)

Release

• 2025-11-07 :HumanSense is accepted by AAAI 2026!
• 2025-08-27 :hearts: We release both the training code and dataset!
• 2025-08-27 :hearts: We released Benchmark and code!
• 2025-08-15 :rocket: We released our paper!

Contents

• Release
• Contents
• HumanSense
• Results
• RUN Your Own Evaluation
• Training Omni Model
• Citation

HumanSense

The evaluation tasks are organized into a four-tier pyramid structure (L1–L4) according to increasing levels of difficulty:

Results

Evaluation Setups: We conduct a comprehensive evaluation of leading Multimodal Large Language Models (MLLMs) with sizes up to 10B, including: (1) Visual LLMs, which represent the most mainstream branch of MLLMs today; (2) Audio LLMs; and (3) Omni-modal LLMs that are natively designed for integrating vision, audio, and text.

RUN Your Own Evaluation

Download the test code from here

Requirements

• Configure the environment required for the model to be tested; the benchmark has no special requirements.

• ffmpeg

  conda activate Modelxx_env (the environment corresponding to the tested model.)
  cd HumanSense-main
  wget https://ffmpeg.org/releases/ffmpeg-4.4.tar.gz
  tar -xvf ffmpeg-4.4.tar.gz
  cd ffmpeg-4.4
  ./configure
  make
  sudo make install
  

Installation

• Download Dataset: Retrieve all necessary files from the folder bench_data in 🤗 HumanSense_Benchmark.

• Decompress Files: Extract the downloaded files and organize them in the ./HumanSense_bench directory as follows:

  HumanSense-main/
  ├── HumanSense_bench/src/data
  │   ├── audios/           
  │   ├── videos/             
  │   ├── HumanSense_AQA.json           
  │   └── HumanSense_VQA.json        
  

Evaluation

• Model Preparation: Prepare your own model for evaluation by following the instructions provided here. This guide will help you set up and configure your model to ensure it is ready for testing against the dataset. Now you can run the benchmark:

• Run and score:

  cd HumanSense-main
  sh HumanSense_bench/eval.sh
  sh HumanSense_bench/eval_audio.sh
  sh HumanSense_bench/score.sh
  

Training Omni Model

We train Qwen25-Omni-7B using 8 x H20 (96G) GPUs

Requirements

# First, configure the environment required to run Qwen25-Omni-7B.
conda activate omni
pip install accelerate
# It's highly recommended to use `[decord]` feature for faster video loading.
pip install qwen-omni-utils[decord] -U


# configure the training requirements
cd HumanSense-main/Open-R1-Video
pip3 install -e ".[dev]"
pip uninstall transformers
unzip transformers-main.zip
cd transformers-main
pip install -e .
cd ..
pip install nvidia-cublas-cu12 -U
pip3 install flash_attn --no-build-isolation
pip uninstall qwen-omni-utils
cd qwen-omni-utils
pip install -e .
cd ..
pip uninstall qwen-vl-utils
cd qwen-vl-utils
pip install -e .
cd ..

pip install qwen-omni-utils[decord] -U
pip install trl==0.14.0
pip install tensorboardX 

Datas Installation

• Download Dataset: Retrieve all necessary files from the folder train_data in 🤗 HumanSense_Benchmark.

• Decompress Files: Extract the downloaded files and organize them in the ./Open-R1-Video directory as follows:

  HumanSense-main/
  ├── Open-R1-Video/data
  │   ├── audios/           
  │   ├── videos/             
  │   ├── merged_video_wo_audio.json
  │   ├── merged_video_audio.json           
  │   └── merged_video_w_audio.json 
  

Training

• Run:

  cd HumanSense-main
  sh Open-R1-Video/framework1/qwen-7b_omni_1video_wo_audio.sh
  sh Open-R1-Video/qwen-7b_omni_2audio.sh
  sh Open-R1-Video/framework2/qwen-7b_omni_3video_w_audio.sh
  

At any stage of training, if the loaded weights do not contain spk_dict.pt, please copy Open-R1-Video/experiments/spk_dict.pt to it.

We release the trained model in 🤗 HumanSense_Omni_Reasoning

• Inference: Modify the model name to "rivideo-omni7B", and subsequently update the loaded weights to the trained ones.

 cd HumanSense-main
 sh HumanSense_bench/eval.sh
 sh HumanSense_bench/eval_audio.sh
 sh HumanSense_bench/score.sh

Citation

If you find our paper and code useful in your research, please consider giving us a star :star: and citing our work :pencil: :)

@article{qin2025humansense,
  title={HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs},
  author={Qin, Zheng and Zheng, Ruobing and Wang, Yabing and Li, Tianqi and Yuan, Yi and Chen, Jingdong and Wang, Le},
  journal={arXiv preprint arXiv:2508.10576},
  year={2025}
}