video-8b-grpo-n16-warm-explore
RLVR post-training of Qwen/Qwen3-VL-8B-Instruct on multiple-choice video QA with fully-async GRPO plus entropy-aware token-dropout exploration, warm-started from the cold-start base RL control at its step 60.
This is the highest offline full-set score of the 8B video campaign: mean_accuracy 0.4942 @ step 40.
It edges the previous best (video-8b-grpo-base @ 80, 0.4918) by +0.24 pt, and beats its own
paired no-exploration control by +0.63 pt at the same step.
Keeper = global_step_40 (peak).
Offline evaluation
Full set = 5,645 rows: VideoMME-v1 2,700 + Video-Holmes 1,837 + PerceptionTest-comp 1,108.
num_failed = 0. mean_accuracy is the unweighted macro mean of the three benchmark accuracies.
| checkpoint | mean_acc | VideoMME | Video-Holmes | PerceptionComp |
|---|---|---|---|---|
| n16 warm+explore @40 (keeper) | 0.4942 | 0.6704 | 0.4774 | 0.3348 |
| n16 warm+explore @20 | 0.4941 | 0.6604 | 0.4807 | 0.3412 |
| n16 no-explore control @40 (paired) | 0.4879 | 0.6570 | 0.4692 | 0.3375 |
| cold-base RL keeper @80 (sibling) | 0.4918 | 0.6581 | 0.4741 | 0.3430 |
| stock Qwen3-VL-8B (zero-shot) | 0.4426 | 0.6404 | 0.4066 | 0.2807 |
Read this before using the number
Three caveats, stated because they change how the result should be interpreted.
- It is a transient peak, not a plateau the run held. Steps 20 and 40 are 0.4941 and 0.4942, then the run declines monotonically to 0.4622 @ 140. Its paired advantage over the control goes +1.3 pt → −1.8 pt across the same span. Later checkpoints of this run are worse than doing nothing.
- The step axis is warm-started. Training resumed from the control's
global_step_60, so "step 40" is roughly gs100 cumulative. It is not comparable on training budget tovideo-8b-grpo-base@ 80, which sits on an absolute cold-start axis. - The margin is inside the campaign's own noise. The top five checkpoints of this campaign span 1.0 pt (0.4942 / 0.4918 / 0.4913 / 0.4890 / 0.4879) on the same corpus and the same 5,645 rows. Across the wider campaign, exploration on video is best described as a wash — see the sibling cards, which reach the same conclusion from different arms.
Training setup
| base | Qwen/Qwen3-VL-8B-Instruct |
| init | warm from cold-start base RL global_step_60 |
| algorithm | GRPO, fully-async (verl fork) |
| corpus | video v3 24f100k, multiple-choice video QA |
| group size | rollout.n = 16, k_explore = 8 |
| exploration | entropy-aware token dropout, tau 0.8 |
| reward | accuracy + format |
| topology | 4 nodes, 2 trainer (dp 16) + 2 rollout (TP 2, 8 vLLM replicas) |
Sibling checkpoints
Same corpus, same 5,645-row evaluation, directly comparable:
ngqtrung/video-8b-grpo-base— cold-start plain GRPO, 0.4918 @80ngqtrung/video-8b-grpo-ppexplore-n16k8— 0.4913 @80ngqtrung/video-8b-grpo-ppexplore— 0.4890 @140ngqtrung/video-8b-grpo-sft770— OMR-SFT-770 warm start, 0.4845 @180
Usage
from transformers import AutoProcessor, AutoModelForImageTextToText
model_id = "ngqtrung/video-8b-grpo-n16-warm-explore"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
The checkpoint is a merged FSDP export: weights, tokenizer and processor are all present, so it loads directly in vLLM as well.