OpenCaption-4B-VL-SFT-v1.0

OpenCaption-4B-VL-SFT-v1.0 is a vision-language captioning model built on top of Qwen/Qwen3-VL-4B-Instruct. It was trained for image captioning and high-quality dense image captioning using a fine-grained mixture of long-form image description traces. The model is designed to produce richer visual understanding than conventional captions by capturing scene composition, object relationships, spatial reasoning, attributes, actions, lighting, background context, and fine visual details.

[!NOTE] This model is an experimental release and may produce unexpected outputs in some scenarios.

GGUF: https://huggingface.co/prithivMLmods/OpenCaption-4B-VL-SFT-v1.0-GGUF

Key Highlights

• Qwen 3-VL Foundation: Built on top of Qwen/Qwen3-VL-4B-Instruct.
• Dense Visual Captioning: Focused on detailed image understanding and long-form image description.
• Fine-Grained Descriptions: Captures object relationships, spatial layout, attributes, actions, lighting, and context.
• Captioning-Oriented Training: Optimized for detailed image caption generation rather than generic short descriptions.
• Research-Focused Release: Intended for experimentation, evaluation, and captioning research.
• Efficient 4B Deployment: Suitable for local inference and research workflows.

Quick Start with Transformers

Install the required packages:

pip install transformers accelerate qwen-vl-utils

Use the model with the following example:

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

# Load the model
model = Qwen3VLForConditionalGeneration.from_pretrained(
    "prithivMLmods/OpenCaption-4B-VL-SFT-v1.0",
    torch_dtype="auto",
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "prithivMLmods/OpenCaption-4B-VL-SFT-v1.0"
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {
                "type": "text",
                "text": "Provide a detailed caption for this image with fine-grained visual details."
            },
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=512
)

generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

print(output_text[0])

System Prompt

OpenCaption-4B-VL-SFT-v1.0 is trained with a specific system prompt. For the best results, use it verbatim.

You are a detailed image captioning assistant.

Structure every caption as follows:

1. Open with one sentence naming the shot type (e.g., eye-level, wide-angle, close-up), the overall setting, and the time of day or lighting condition.

2. Break the rest of the description into thematic sections, each introduced by a bold markdown header ending in a colon (e.g., **The Subject:**, **The Background:**, **Atmosphere & Lighting:**), chosen to fit what is actually in the image.

3. Within each section, use bullet points to list specific, concrete details, including positions, colors, textures, materials, actions, spatial relationships, and any legible text or fine-grained visual elements. If a section covers multiple distinct areas of the frame, introduce each with its own nested bold sub-label ending in a colon (e.g., **Foreground Right:**, **Background:**) before its bullet points.

4. Close with a short, unheaded paragraph beginning with "In summary," that ties the scene together and conveys its overall mood or narrative.

Use precise, sensory, and fluent language throughout. Describe only what is visible in the image, avoid speculation or unsupported inferences, and do not use emojis.

Training Details

Intended Use

• Image Captioning: Generating clear and detailed captions for images.
• Dense Image Understanding: Describing scenes with richer visual context and fine-grained detail.
• Visual Reasoning: Explaining object placement, interactions, and relationships in an image.
• Research and Evaluation: Studying caption quality, visual grounding, and multimodal fine-tuning.
• Local Deployment: Running a compact 4B vision-language captioning model for experimentation.

Limitations

• Experimental Model: Output quality may vary across domains and image styles.
• Caption Biases: Performance depends on the diversity and quality of the training traces.
• Hallucinations: Like other multimodal models, the model may occasionally describe details that are not present.
• Not a Ground-Truth Detector: The model is designed for captioning and visual description, not strict object verification.

Acknowledgements

• Qwen/Qwen3-VL-4B-Instruct: Base model used for this project.
• Transformers: Transformers provides state-of-the-art machine learning models for text, computer vision, audio, video, and multimodal tasks, supporting both inference and training.
• TRL - Transformers Reinforcement Learning: TRL is a full stack library providing tools to train transformer language models with methods including Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Reward Modeling, and more.