Qwen3-VL-8B · Video-MC · GRPO + ppexplore (τ0.95, cold-start)
RLVR post-training of Qwen/Qwen3-VL-8B-Instruct on multiple-choice video QA with fully-async GRPO and an entropy-aware token-dropout exploration ("ppexplore", τ=0.95) at the rollout stage, cold-started from the stock 8B-Instruct. This is experiment_name=grpo_video_4node_full_v3_24f100k_8b_base_ppexplore_v1, global_step_140 (the keeper).
On video, exploration is a WASH: cold-start ppexplore (τ0.95) peaks at offline full-set val mean_accuracy 0.4890 @ step 140 — statistically tied with the cold-start base (0.4918) and SFT-770 warmstart (0.4845). The token-dropout exploration that won +4.6 pt on OMR delivers no gain on video; this run sits in the same ~0.485 dead-heat band.
Results
Offline full-set eval (VideoMME-v1 2700 + PerceptionComp 1108 + Video-Holmes 1837 = 5645 rows), scored with the repo's vero compute_score. mean = macro-mean of the 3 bench accuracies.
Keeper = global_step_140 (peak):
| metric | mean | videomme | holmes | perceptioncomp |
|---|---|---|---|---|
| ppexplore τ0.95 @140 (keeper) | 0.4890 | 0.656 | 0.465 | 0.346 |
| cold-base RL keeper @80 (sibling) | 0.4918 | 0.658 | 0.474 | 0.343 |
| SFT-770 RL keeper @180 (sibling) | 0.4845 | 0.657 | 0.451 | 0.345 |
| stock Qwen3-VL-8B ckpt-0 (zero-shot) | 0.4444 | 0.6426 | 0.4143 | 0.2762 |
Trajectory (mean): 0.451 @20 → 0.470 @40 → 0.477 @60 → 0.488 @100 → 0.489 @140 → faded late (0.477 @200, 0.480 @260). Ran to ~step 780 (near 2 epochs); further training brought no val gain — same post-peak fade as every other video run. ppexplore's lower train reward did not translate to higher val.
Training
- Base model: stock
Qwen/Qwen3-VL-8B-Instruct(cold-start,resume_mode=auto). - Framework: fork of volcengine/verl —
ngquangtrung57/verl@videorl-mods. Fully-async GRPO: FSDP2 trainer + vLLM rollouter. - Warm start: none (cold-start base).
- Reward: dapo-style
score = 0.8·accuracy + 0.2·format(FORMAT_WEIGHT=0.2,FORMAT_MIN_THINK_CHARS=100). No KL penalty. - Topology: 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
- Batch:
ppo_mini_batch_size=16×require_batches=4×rollout.n=8= 512 trajectories/step. - Data:
GROUP_VIDEO_TRAIN_MC_24F100K(5 video-MC parquets, 24 frames / 100k pixels). - Optim / seq: lr
1e-6, warmup 25 steps;total_epochs=2; clip_ratio 0.2 / 0.3 (clip_c=10.0);max_prompt_length=17408,max_response_length=16384;enforce_eager=true;gpu_memory_utilization=0.75; staleness 0.4. - Exploration block (entropy-aware token-dropout at rollout; byte-identical to the OMR-winner τ0.95 block):
| key | value |
|---|---|
enable | true |
trigger_mode | high |
top_prob_threshold (τ) | 0.95 |
k_explore | 4 (of n=8 rollouts explore; 4 stay clean) |
prompt_exploration_prob | 0.5 |
deterministic | true |
perturb_prob | 1.0 |
mask_from_loss | true |
drop_top_k | 1 |
restrict_to_think_region | true |
selection_seed | 42 |
- Validation: inline val OFF (
test_freq=10000); video val is offline full-set eval on a dedicated 8×H100 node (vLLM TP1), every 20 fit-steps. - Train metrics: ~219 s/step; final reward 0.858; final response_length ~164 tok; 780 steps trained (~2 epochs).
W&B
Project verl_fully_async (entity quangtrung5705-nanyang-technological-university-singapore). Train metrics only — video val is offline, not on W&B:
https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/88lmybpk
Intended use / limitations
Research checkpoint — the cold-start exploration arm of an 8B video study. Headline finding: on video, exploration is a documented dead-heat — the OMR-winning recipe applied to video does not break the ~0.485 full-set val ceiling (this run, base, SFT-warmstart, and several further exploration ablations all land ~0.485-0.49). Useful as an A/B reference, not as a "better video model" — prefer video-8b-grpo-base (0.4918) if you just want the best keeper. Multiple-choice video QA, <think>…</think> then-answer format. No safety/RLHF alignment beyond the base.
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "ngqtrung/video-8b-grpo-ppexplore"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)
messages = [{
"role": "user",
"content": [
{"type": "video", "video": "clip.mp4"},
{"type": "text", "text": "Answer the multiple-choice question. Reason inside <think>...</think>, then give the final letter."},
],
}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
Citation / lineage
- Base model: Qwen3-VL-8B-Instruct (Qwen team). Inherits the Qwen3-VL license — review the base model's terms; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
- Framework: verl (volcengine/verl), fork
ngquangtrung57/verl@videorl-mods; fully-async GRPO (FSDP2 + vLLM). - Method: entropy-aware token-dropout exploration ("ppexplore", τ=0.95) at the rollout stage. Part of a controlled OMR/Video exploration study on Qwen3-VL-8B (
docs/experiments_summary_8b.md).