SAGA_Qwen3-8B
SAGA-finetuned Qwen3-VL-8B model for short-drama video understanding, trained on the M-Drama dataset (SFT + RL).
This model is introduced in Beyond Sparse Rewards: A New Benchmark and Structure-Aware Graph Alignment for Micro-Drama Understanding.
Code: github.com/qyx1121/MDrama_SAGA
Important: This model is trained to answer with a fixed three-block structure (
<caption>→<think>→<answer>). Always pass the system prompt below, otherwise the output format (and therefore any downstream parsing) will not match the reported results.
Prompts
System prompt (required, shared by both tasks)
You are an expert AI assistant specializing in video analysis. Your purpose is to provide clear, detailed, and accurate analysis of video content based on the user's request.
When you are faced with the task of video description, your response should describe the video's content chronologically, identify key highlights, and provide a concise summary. Structure your output strictly in the following format:
<caption>
A detailed, chronological description of the video's events and scenes from start to finish.
</caption>
<think>
A bulleted list that capture the story's highlights and narrative hooks.
</think>
<answer>
A brief, high-level summary of the entire video.
</answer>
When you are responsible for answering questions, your response must be well-reasoned and detailed. First, provide a full chronological description of the video to establish context. Then, use an internal monologue to reason through the evidence. Finally, provide the answer. Structure your output strictly in the following format:
<caption>
A detailed, chronological description of the video's events and scenes from start to finish.
</caption>
<think>
This is your internal monologue. Extract useful information from the context, think step-by-step, connect the evidence, and formulate your reasoning process.
</think>
<answer>
Based on your reasoning and caption, provide a comprehensive and direct answer to the user's question.
</answer>
User prompt — Summary / video description
<video>
Please describe the video and analyze the plot in detail.
User prompt — QA
Open-ended:
{question}
Multiple-choice (options are appended to the question, separated by ; ):
{question}
Options: [A. {option A}; B. {option B}; C. {option C}; D. {option D}]
For multiple-choice questions the model is expected to emit only the option letter (e.g. A) inside <answer>.
Usage
Video sampling args used in the paper's evaluation: fps=2.0, max_frames=192,
max_pixels=128*28*28, min_pixels=64*28*28 (the summary script omits min_pixels;
results are equivalent in practice).
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from qwen_vl_utils import process_vision_info
MODEL_ID = "yixin1121/SAGA_Qwen3-8B"
SYSTEM_PROMPT = """You are an expert AI assistant specializing in video analysis. Your purpose is to provide clear, detailed, and accurate analysis of video content based on the user's request.
When you are faced with the task of video description, your response should describe the video's content chronologically, identify key highlights, and provide a concise summary. Structure your output strictly in the following format:
<caption>
A detailed, chronological description of the video's events and scenes from start to finish.
</caption>
<think>
A bulleted list that capture the story's highlights and narrative hooks.
</think>
<answer>
A brief, high-level summary of the entire video.
</answer>
When you are responsible for answering questions, your response must be well-reasoned and detailed. First, provide a full chronological description of the video to establish context. Then, use an internal monologue to reason through the evidence. Finally, provide the answer. Structure your output strictly in the following format:
<caption>
A detailed, chronological description of the video's events and scenes from start to finish.
</caption>
<think>
This is your internal monologue. Extract useful information from the context, think step-by-step, connect the evidence, and formulate your reasoning process.
</think>
<answer>
Based on your reasoning and caption, provide a comprehensive and direct answer to the user's question.
</answer>"""
SUMMARY_PROMPT = "<video>\nPlease describe the video and analyze the plot in detail."
model = AutoModelForImageTextToText.from_pretrained(
MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto",
attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(MODEL_ID)
processor.video_processor.do_sample_frames = False
def build_messages(video_path: str, user_prompt: str):
return [
{"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
{
"role": "user",
"content": [
{
"type": "video",
"video": video_path,
"fps": 2.0,
"max_frames": 192,
"max_pixels": 128 * 28 * 28,
"min_pixels": 64 * 28 * 28,
},
{"type": "text", "text": user_prompt},
],
},
]
def generate(video_path: str, user_prompt: str, max_new_tokens: int = 2048) -> str:
messages = build_messages(video_path, user_prompt)
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages, return_video_metadata=True)
videos = [v[0] for v in video_inputs]
video_metadata = [v[1] for v in video_inputs]
inputs = processor(
text=[text],
images=None,
videos=videos,
video_metadata=video_metadata,
padding=True,
return_tensors="pt",
do_sample_frames=False,
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(**inputs, max_new_tokens=max_new_tokens)
generated_ids_trimmed = [
out_ids[len(in_ids):]
for in_ids, out_ids in zip(inputs["input_ids"], generated_ids)
]
return processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
Example 1 — Video summarization
out = generate("path/to/clip.mp4", SUMMARY_PROMPT)
print(out)
Expected output shape:
<caption>
The clip opens in a brightly lit banquet hall where Victoria is being mocked by two guests ...
</caption>
<think>
- Victoria is publicly humiliated, establishing her as the underdog.
- Nicole's intervention reframes the power dynamic ...
</think>
<answer>
A bullied heiress is publicly defended at a banquet, setting up a reversal-of-status arc ...
</answer>
Example 2 — Multiple-choice QA
question = "Why does Nicole push past the woman in red and head straight toward Victoria?"
candidates = [
"A. She is angered by their contempt for Victoria",
"B. She wants to thank Victoria for finding her",
"C. She is going to get another wine glass",
"D. She intends to leave the banquet early",
]
prompt = question + "\nOptions: [" + "; ".join(candidates) + "]"
out = generate("path/to/clip.mp4", prompt)
# parse the final answer
from bs4 import BeautifulSoup
pred = BeautifulSoup(out, "html.parser").find("answer")
pred = pred.get_text(strip=True) if pred else ""
print(pred) # -> "A"
Example 3 — Open-ended QA
out = generate(
"path/to/clip.mp4",
'In the video, who is revealed to be "Miss Lewis," the Lewis Group heiress '
'who owns the entire brand?',
)
print(out) # <caption>...</caption><think>...</think><answer>Laura</answer>
Output format
| Task | Parsing |
|---|---|
| Summary | Use the full text, or extract caption (detailed description) and answer (high-level summary) |
| Open-ended QA | Extract the text inside <answer> |
| Multiple-choice QA | Extract the text inside <answer> and compare to the gold option letter |
Training
- Base model: Qwen3-VL-8B-Instruct
- Training data: M-Drama — 32,361 QA annotations over 8,102 short-drama video clips
- Pipeline: SFT followed by RL (DAPO-style, with graph-structure and semantic rewards)
- Decoding used for reported numbers: greedy,
max_new_tokens=2048, 192 frames at 2 fps
License
CC BY-NC 4.0. This model is intended for research purposes only.