⚠️ Disclaimer & Usage Policy

NOTICE: This model has been modified using Directional Activation Abliteration to remove standard safety refusal vectors and alignment guardrails present in the base model.

1. Research & Educational Purpose: This model is released primarily for AI safety researchers, ethics auditors, and developers studying alignment, refusal mechanics, and uncensored multimodal video understanding.
2. User Responsibility: By downloading and using this model, you acknowledge that it may generate unfiltered, unbiased, or explicit descriptions of user-generated video and audio content. The creators, modifier, and host platform assume no liability or responsibility for any misuse, unsafe output, or harmful content generated by end-users.
3. Non-Endorsement: The removal of refusal guardrails does not imply endorsement of harmful, illegal, or unethical content. Users must comply with local laws and regulations regarding AI-generated content.

🔓 UGC-VideoCaptioner-Abliterated (Uncensored 3B)

UGC-VideoCaptioner-Abliterated is an uncensored, refusal-free version of openinterx/UGC-VideoCaptioner (a 3B omnimodal video detail captioning model built on Qwen2.5-Omni).

Using Directional Activation Abliteration, the safety refusal vectors within the core LLM "Thinker" backbone were calculated and mathematically projected out of the model's weight matrices. This removes refusal behaviors without fine-tuning, retraining, or degrading the model's vision-audio captioning capabilities.

🛠️ Technical Engineering Details

Traditional fine-tuning to remove safety guardrails often leads to catastrophic forgetting or degradation of multimodal reasoning. Instead, this model was uncensored using In-Place Weight Orthogonalization:

1. Residual Stream Activation Caching

We run paired batches of harmful instructions ($D_{harmful}$) and benign instructions ($D_{harmless}$) through the LLM text backbone (thinker.model). Forward hooks capture the hidden states in the residual stream at the final token position across all transformer layers:

$$\mu_{harmful}^{(l)} = \frac{1}{N} \sum_{i=1}^{N} h_{i, \text{last}}^{(l)} (D_{harmful})$$

$$\mu_{harmless}^{(l)} = \frac{1}{N} \sum_{i=1}^{N} h_{i, \text{last}}^{(l)} (D_{harmless})$$

2. Refusal Direction Calculation

The mean difference vector $\hat{r}^{(l)}$ represents the specific direction in representation space that triggers safety refusals:

$$\hat{r}^{(l)} = \frac{\mu_{harmful}^{(l)} - \mu_{harmless}^{(l)}}{|\mu_{harmful}^{(l)} - \mu_{harmless}^{(l)}|_2}$$

3. Orthogonal Weight Projection

The optimal layer candidate ($\text{Layer } 9$) is selected. To prevent the model from activating the refusal subspace during generation, we project out $\hat{r}$ from the Embedding matrix ($W_E$), Attention Output Projections ($W_O$), and MLP Down Projections ($W_{down}$) across all transformer layers:

$$W_{E}' = W_E - (W_E \cdot \hat{r}) \hat{r}^T$$

$$W_{O}' = W_O - \hat{r} \cdot (\hat{r}^T W_O)$$

$$W_{down}' = W_{down} - \hat{r} \cdot (\hat{r}^T W_{down})$$

4. Multimodal Architecture Preservation

Because directional ablation is performed strictly on the internal linear projections of thinker.model, all surrounding multimodal encoders remain 100% untouched:

• 🎬 thinker.visual: All 32 vision transformer blocks + patch embeddings.
• 🎵 thinker.audio_tower: All 32 audio transformer layers + convolutional layers.
• 🗣️ talker: Speech generation head & projector modules.

UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks

UGC-VideoCaptioner is a 3B-parameter captioning model distilled from Gemini-2.5 Flash, specifically designed for detailed, omnimodal captioning of short-form, user-generated videos (UGC). It addresses the crucial role of audio in conjunction with visual content, which is often overlooked in existing video captioning models.

Abstract

Real-world user-generated videos, especially on platforms like TikTok, often feature rich and intertwined audio visual content. However, existing video captioning benchmarks and models remain predominantly visual centric, overlooking the crucial role of audio in conveying scene dynamics, speaker intent, and narrative context. This lack of omni datasets and lightweight, capable models hampers progress in fine grained, multimodal video understanding. To address these challenges, we introduce UGC-VideoCap, a new benchmark and model framework specifically designed for detailed omnimodal captioning of short form user-generated videos. Unlike prior datasets, UGC-VideoCap emphasizes balanced integration of audio and visual modalities, featuring 1000 TikTok videos annotated through a structured three stage human-in-the-loop pipeline covering audio only, visual only, and joint audio visual semantics. The benchmark also includes 4000 carefully crafted QA pairs probing both unimodal and cross modal understanding. Alongside the dataset, we propose UGC-VideoCaptioner(3B), a 3B parameter captioning model distilled from Gemini 2.5 Flash. Using a novel two-stage training strategy supervised fine tuning followed by Group Relative Policy Optimization (GRPO), our approach enables efficient adaptation from limited data while maintaining competitive performance. Together, our benchmark and model offer a high-quality foundation and a data-efficient solution for advancing omnimodal video captioning in unconstrained real-world UGC settings.

Benchmark Results

Model Zoom

Quick Start

You can use this model with the transformers library. Below is a quick example demonstrating how to perform inference. Please note that for full video processing capabilities, you might need to install decord and refer to the official GitHub repository for detailed video handling steps, especially if AutoProcessor doesn't directly handle video file paths for complex scenarios.

Environment Setup

pip install transformers torch decord soundfile qwen_omni_utils

Inference

import soundfile as sf

from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info

model = Qwen2_5OmniForConditionalGeneration.from_pretrained("openinterx/UGC-VideoCaptioner", torch_dtype="auto", device_map="auto")

# We recommend enabling flash_attention_2 for better acceleration and memory saving.
# model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
#     "Qwen/Qwen2.5-Omni-3B",
#     torch_dtype="auto",
#     device_map="auto",
#     attn_implementation="flash_attention_2",
# )

processor = Qwen2_5OmniProcessor.from_pretrained("openinterx/UGC-VideoCaptioner")

# Example video path (replace with your actual video file path)
video_path = "path/to/your/video.mp4" 

# Define the detailed captioning prompt
prompt_text = (
    "You are given a short video with both audio and visual content. Write a detailed and coherent paragraph "
    "that naturally integrates all modalities. Your description should include: (1) the primary scene and "
    "background setting; (2) key characters or objects and their actions or interactions; (3) significant "
    "audio cues such as voices, background music, sound effects, and their emotional tone; (4) any on-screen "
    "text (OCR) and its role in the video context; and (5) the overall theme or purpose of the video. "
    "Ensure the output is a fluent and objective paragraph, not a bullet-point list, and captures the video's "
    "content in a human-like, narrative style."
)

# Prepare messages in the chat template format
messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": video_path}, # Pass video path
            {"type": "text", "text": prompt_text},
        ],
    }
]


# set use audio in video
USE_AUDIO_IN_VIDEO = True

# Preparation for inference
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=USE_AUDIO_IN_VIDEO)
inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", padding=True, use_audio_in_video=USE_AUDIO_IN_VIDEO)
inputs = inputs.to(model.device).to(model.dtype)

# Inference: Generation of the output text and audio
text_ids, audio = model.generate(**inputs, use_audio_in_video=USE_AUDIO_IN_VIDEO)

text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(text)
sf.write(
    "output.wav",
    audio.reshape(-1).detach().cpu().numpy(),
    samplerate=24000,
)

vllm inference

# pip install vllm
# pip install transformers==4.52.3


import os
import json
import re
from tqdm import tqdm
from vllm import LLM, SamplingParams
from vllm.assets.video import VideoAsset
from vllm.utils import FlexibleArgumentParser

VIDEO_DIR = "/workspace/benchmark/video"
OUTPUT_JSONL = "/workspace/benchmark/omni_vllm_sft_result_same_parameter.jsonl"
USE_AUDIO_IN_VIDEO = True
MAX_RETRY = 3

# Ensure output file exists
def ensure_output_file(path: str):
    if not os.path.exists(path):
        open(path, "w", encoding="utf-8").close()

# Load processed video IDs to skip

def load_processed_ids(jsonl_path: str) -> set[str]:
    processed = set()
    with open(jsonl_path, "r", encoding="utf-8") as fin:
        for line in fin:
            try:
                data = json.loads(line)
                vid = data.get("video_id")
                if vid:
                    processed.add(vid)
            except json.JSONDecodeError:
                continue
    return processed

# Regex to verify level tag at end of caption 
# 没有level
LEVEL_PATTERN = re.compile(r"<level>[A-F]</level>\s*$")

PROMPT_TEMPLATE = (
    f"<|im_start|>system\n" +
    "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech. Please make sure that the content within the answer is long and detailed enough." +
    "<|im_end|>\n"
    "<|im_start|>user\n<|vision_bos|><|VIDEO|><|vision_eos|>"
    "You are given a short video with both audio and visual content. Write a detailed and coherent paragraph that naturally integrates all modalities. "
    "Your description should include: (1) the primary scene and background setting; (2) key characters or objects and their actions or interactions; "
    "(3) significant audio cues such as voices, background music, sound effects, and their emotional tone; "
    "(4) any on-screen text (OCR) and its role in the video context; and (5) the overall theme or purpose of the video. "
    "Ensure the output is a fluent and objective paragraph, not a bullet-point list, and captures the video's content in a human-like, narrative style. <|im_end|>\n"
    "<|im_start|>assistant\n"
)


def process_video_folder(model_name: str, seed: int = None):
    ensure_output_file(OUTPUT_JSONL)
    video_files = sorted(f for f in os.listdir(VIDEO_DIR) if f.lower().endswith(".mp4"))
    processed_ids = load_processed_ids(OUTPUT_JSONL)

    llm = LLM(
        model=model_name,
        max_model_len=20000,
        max_num_seqs=5,
        limit_mm_per_prompt={"video": 1, "audio": 1},
        seed=seed,
    )
    sampling_params = SamplingParams(temperature=0.2, max_tokens=1024)

    with open(OUTPUT_JSONL, "a", encoding="utf-8") as fout:
        for fname in tqdm(video_files, desc="Processing videos"):
            video_id = os.path.splitext(fname)[0]
            if video_id in processed_ids:
                print(f"[Skip] {fname} already processed, skipping.")
                continue

            fpath = os.path.join(VIDEO_DIR, fname)
            valid_caption = None
            try:
                video_asset = VideoAsset(path=fpath, num_frames=32)
                audio = video_asset.get_audio(sampling_rate=16000)

                inputs = {
                    "prompt": PROMPT_TEMPLATE,
                    "multi_modal_data": {"video": video_asset.np_ndarrays, "audio": audio},
                    "mm_processor_kwargs": {"use_audio_in_video": USE_AUDIO_IN_VIDEO},
                }

                for attempt in range(MAX_RETRY):
                    outputs = llm.generate(inputs, sampling_params=sampling_params)
                    text = outputs[0].outputs[0].text.strip()
                    if text and LEVEL_PATTERN.search(text):
                        valid_caption = text
                        break
                    else:
                        print(f"[Retry] Attempt {attempt+1} for {fname} did not end with level tag, retrying...")

                if not valid_caption:
                    print(f"[Warning] {fname} failed to get valid level tag after {MAX_RETRY} attempts, skipping.")
                    continue

                fout.write(json.dumps({"video_id": video_id, "caption": valid_caption}, ensure_ascii=False) + "\n")
                fout.flush()
                processed_ids.add(video_id)

            except Exception as e:
                print(f"[Error] Failed to process {fname}: {e}")
                continue

    print(f"✅ Done! Processed videos with skipping and level validation. Output written to {OUTPUT_JSONL}")


def parse_args():
    parser = FlexibleArgumentParser(description="Batch inference for a folder of videos using Qwen2.5-Omni + vLLM.")
    parser.add_argument("--model-name", type=str, default="/workspace/output_model/tiktok_caption/omni_sft_20k_level/v1-20250701-150049/checkpoint-2404-merged", help="Model path or name.")
    parser.add_argument("--seed", type=int, default=42, help="Random seed for reproducibility.")
    return parser.parse_args()


if __name__ == "__main__":
    args = parse_args()
    process_video_folder(args.model_name, args.seed)

Evaluation

Final Caption prompt (for inference)

prompt = "You are given a short video with both audio and visual content. Write a detailed and coherent paragraph that naturally integrates all modalities. "
"Your description should include: (1) the primary scene and background setting; (2) key characters or objects and their actions or interactions; "
"(3) significant audio cues such as voices, background music, sound effects, and their emotional tone; "
"(4) any on-screen text (OCR) and its role in the video context; and (5) the overall theme or purpose of the video. "
"Ensure the output is a fluent and objective paragraph, not a bullet-point list, and captures the video's content in a human-like, narrative style.

Score

Scores are judged by GPT-4o-2024-08-06.

python eval_caption.py

💻 Full Colab Reproduction Notebook

You can reproduce this abliteration process directly in Google Colab using the script below.

# ==========================================
# CELL 1: Environment Setup & Dataset Loading
# ==========================================
!pip install -qqq transformers==4.52.3 soundfile transformers_stream_generator tiktoken einops jaxtyping accelerate huggingface_hub --progress-bar off

import torch
import gc
import os
from datasets import load_dataset
from tqdm import tqdm
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2TokenizerFast, AutoProcessor
from huggingface_hub import HfApi, snapshot_download
import transformers.modeling_rope_utils as rope_utils

torch.set_grad_enabled(False)

def reformat_texts(texts):
    return [[{"role": "user", "content": text}] for text in texts]

print("Loading activation pair datasets...")
harmful_dataset = load_dataset('mlabonne/harmful_behaviors')
harmless_dataset = load_dataset('mlabonne/harmless_alpaca')

harmful_train = reformat_texts(harmful_dataset['train']['text'])
harmless_train = reformat_texts(harmless_dataset['train']['text'])

# ==========================================
# CELL 2: Download Base Model & RoPE Patch
# ==========================================
LOCAL_PATH = "./UGC_Repo"

print("Downloading base repository...")
snapshot_download(
    repo_id="openinterx/UGC-VideoCaptioner",
    local_dir=LOCAL_PATH,
    local_dir_use_symlinks=False
)

# Patch RoPE initialization compatibility for transformers >= 4.50
if "default" not in rope_utils.ROPE_INIT_FUNCTIONS:
    def _init_default_rope(config, device=None, seq_len=None):
        dim = getattr(config, "head_dim", config.hidden_size // getattr(config, "num_attention_heads", 1))
        base = getattr(config, "rope_theta", 10000.0)
        inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=device).float() / dim))
        return inv_freq, 1.0
    rope_utils.ROPE_INIT_FUNCTIONS["default"] = _init_default_rope

print("Loading base model weights into GPU...")
hf_model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    LOCAL_PATH,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

tokenizer = Qwen2TokenizerFast.from_pretrained(LOCAL_PATH)
tokenizer.padding_side = 'left'
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# ==========================================
# CELL 3: Directional Abliteration
# ==========================================
thinker_model = hf_model.thinker.model if hasattr(hf_model, "thinker") else hf_model.model
layers = thinker_model.layers
num_layers = len(layers)

n_inst = min(128, len(harmful_train), len(harmless_train))
batch_size = 16

harmful_acts = {i: [] for i in range(num_layers)}
harmless_acts = {i: [] for i in range(num_layers)}

def get_activation(storage, layer_idx):
    def hook(module, input, output):
        hidden = output[0] if isinstance(output, tuple) else output
        storage[layer_idx].append(hidden[:, -1, :].detach().cpu())
    return hook

def tokenize_batch(texts):
    out = tokenizer.apply_chat_template(
        texts, padding=True, truncation=False, return_tensors="pt", add_generation_prompt=True
    )
    if isinstance(out, dict) or hasattr(out, "keys"):
        return out["input_ids"].to(hf_model.device)
    return out.to(hf_model.device)

# 1. Collect Harmful Activations
for i in tqdm(range(0, n_inst, batch_size), desc="Caching Harmful"):
    input_ids = tokenize_batch(harmful_train[i:i+batch_size])
    hooks = [layer.register_forward_hook(get_activation(harmful_acts, idx)) for idx, layer in enumerate(layers)]
    hf_model.thinker(input_ids=input_ids)
    for h in hooks: h.remove()
    del input_ids
    gc.collect()
    torch.cuda.empty_cache()

# 2. Collect Harmless Activations
for i in tqdm(range(0, n_inst, batch_size), desc="Caching Harmless"):
    input_ids = tokenize_batch(harmless_train[i:i+batch_size])
    hooks = [layer.register_forward_hook(get_activation(harmless_acts, idx)) for idx, layer in enumerate(layers)]
    hf_model.thinker(input_ids=input_ids)
    for h in hooks: h.remove()
    del input_ids
    gc.collect()
    torch.cuda.empty_cache()

# 3. Compute Refusal Vector
refusal_dirs = []
for idx in range(num_layers):
    h_mean = torch.cat(harmful_acts[idx], dim=0).mean(dim=0)
    hl_mean = torch.cat(harmless_acts[idx], dim=0).mean(dim=0)
    r_dir = h_mean - hl_mean
    r_dir = r_dir / r_dir.norm()
    refusal_dirs.append(r_dir)

# 4. Orthogonalize Weights
LAYER_CANDIDATE = 9
refusal_vec = refusal_dirs[LAYER_CANDIDATE].to(hf_model.device, dtype=torch.bfloat16)

# Abliterate Embeddings
embed_weight = thinker_model.embed_tokens.weight
proj_embed = (embed_weight @ refusal_vec).unsqueeze(1) * refusal_vec
embed_weight.data -= proj_embed

# Abliterate Output Projections across all layers
for layer in tqdm(layers, desc="Abliterating Layers"):
    w_o = layer.self_attn.o_proj.weight
    layer.self_attn.o_proj.weight.data -= torch.outer(refusal_vec, refusal_vec @ w_o)
    w_d = layer.mlp.down_proj.weight
    layer.mlp.down_proj.weight.data -= torch.outer(refusal_vec, refusal_vec @ w_d)

print("Abliteration complete!")

# ==========================================
# CELL 4: Export & Upload to Hugging Face
# ==========================================
HF_TOKEN = "YOUR_HF_TOKEN_HERE" # Replace with your Hugging Face write token
HF_REPO_NAME = "MahouOfficial/UGC-VideoCaptioner-Abliterated"

os.environ["HF_TOKEN"] = HF_TOKEN
api = HfApi(token=HF_TOKEN)
api.create_repo(repo_id=HF_REPO_NAME, exist_ok=True, private=False)

TEMP_DIR = "./temp_export"
os.makedirs(TEMP_DIR, exist_ok=True)

# Save modified weights & tokenizer locally
hf_model.save_pretrained(TEMP_DIR, safe_serialization=True)
tokenizer.save_pretrained(TEMP_DIR)

# Copy missing processor assets
import shutil
for item in os.listdir(LOCAL_PATH):
    src_path = os.path.join(LOCAL_PATH, item)
    dst_path = os.path.join(TEMP_DIR, item)
    if os.path.isfile(src_path) and not os.path.exists(dst_path):
        shutil.copy2(src_path, dst_path)

# Upload consolidated repository
api.upload_folder(
    folder_path=TEMP_DIR,
    repo_id=HF_REPO_NAME,
    token=HF_TOKEN
)
print(f"Model successfully uploaded to https://huggingface.co/{HF_REPO_NAME}")

📜 Original Model Citation

If you use this model or dataset in your research, please cite the original authors:

@article{wu2025ugc,
  title={UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks},
  author={Wu, Peiran and Liu, Yunze and Zhu, Zhengdong and Zhou, Enmin and Shen, Shawn},
  journal={arXiv preprint arXiv:2507.11336},
  year={2025}
}