About Dataset
You need to first download the HumanCaption from our huggingface and then apply for access to the original Laion-face dataset by completing the required agreement (github). Once approved, refer to the information available on HuggingFace to obtain the corresponding image-text pairs.
[25/06/09] 🤗The Original Images, are Released Completing the Agreement
DEMO
Introduction
Human-related vision and language tasks are widely applied across various social scenarios. The latest studies demonstrate that the large vision-language model can enhance the performance of various downstream tasks in visual-language understanding. Since, models in the general domain often not perform well in the specialized field. In this study, we train a domain-specific Large Language-Vision model, Human-LLaVA, which aim to construct an unified multimodal Language-Vision Model for Human-related tasks.
Specifically, (1) we first construct a large-scale and high-quality human-related image-text (caption) dataset extracted from Internet for domain-specific alignment in the first stage (Coming soon); (2) we also propose to construct a multi-granularity caption for human-related images (Coming soon), including human face, human body, and whole image, thereby fine-tuning a large language model. Lastly, we evaluate our model on a series of downstream tasks, our Human-LLaVA achieved the best overall performance among multimodal models of similar scale. In particular, it exhibits the best performance in a series of human-related tasks, significantly surpassing similar models and ChatGPT-4o. We believe that the Huaman-LLaVA model and a series of datasets presented in this work can promote research in related fields.
Result
human-llava has a good performance in both general and special fields
News and Update 🔥🔥🔥
- Oct.23, 2024. 🤗HumanCaption-HQ-311K, is released!👏👏👏
- Sep.12, 2024. 🤗HumanCaption-10M, is released!👏👏👏
- Sep.8, 2024. 🤗HumanVLM, is released!👏👏👏
🤗 Transformers
To use Human-LLaVA for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, please make sure that you are using latest code.
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForPreTraining
model_id = "OpenFace-CQUPT/Human_LLaVA"
cuda = 0
model = AutoModelForPreTraining.from_pretrained("OpenFace-CQUPT/Human_LLaVA", torch_dtype=torch.float16).to(cuda)
processor = AutoProcessor.from_pretrained(model_id,trust_remote_code=True)
text = "Please describe this picture"
prompt = "USER: <image>\n" + text + "\nASSISTANT:"
image_file = "./test1.jpg"
raw_image = Image.open(image_file)
# raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to(cuda, torch.float16)
output = model.generate(**inputs, max_new_tokens=400, do_sample=False)
predict = processor.decode(output[0][:], skip_special_tokens=True)
print(predict)
Our training code have been released publicly on github.ddw2AIGROUP2CQUPT/Human-LLaVA-8B(github.com)
Get the Dataset
Dataset Example
Domain Alignment Stage
HumanCaption-10M(self construct): is released!
Instruction Tuning Stage
All public data sets have been filtered, and we will consider publishing all processed text in the future
HumanCaption-HQ(self construct): is released!
FaceCaptionA(self construct): is released!
CelebA: https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
ShareGPT4V:https://github.com/InternLM/InternLM-XComposer/blob/main/projects/ShareGPT4V/docs/Data.md
LLaVA-Instruct_zh : https://huggingface.co/datasets/openbmb/llava_zh
verified_ref3rec: https://huggingface.co/datasets/lucasjin/refcoco/blob/main/ref3rec.json
verified_ref3reg: https://huggingface.co/datasets/lucasjin/refcoco/blob/main/ref3rec.json
verified_shikra: https://github.com/shikras/shikra
Citation
Dai D, Xu L, Li Y, et al. Humanvlm: Foundation for human-scene vision-language model[J]. Information Fusion, 2025: 103271.
contact
mailto: S230201133@stu.cqupt.edu.cn or dw_dai@163.com