ReFoCUS-0.3B

ReFoCUS (Reinforcement-guided Frame Optimization for Contextual UnderStanding) is a lightweight frame-selection model for long-video understanding. Given a video and a question, it identifies the frames that are truly essential for answering the question and hands them to a downstream Video-LLM, replacing uniform sampling with query-conditioned visual evidence.

Paper: ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

Hosu Lee*, Junho Kim*, Hyunjun Kim, Yong Man Ro

KAIST · UIUC

What is ReFoCUS?

ReFoCUS is a framework for finding the visual evidence scattered along a video's temporal axis that is truly essential for answering a given query. The evidence needed to answer a question is usually sparse and unevenly distributed, yet most video-LLMs still consume a fixed set of uniformly sampled frames, and heuristic or retrieval-based selection is never jointly optimized with the model's own reasoning, so it often conflates raw visual dynamics with true semantic relevance.

ReFoCUS is the first framework to integrate online policy-gradient reinforcement learning into frame-level optimization for video-LLMs. Instead of aligning the textual outputs of an LMM with preferences, it optimizes the visual inputs the model attends to:

• Policy model (this model): reads the dense video together with the question and autoregressively picks the frames that best support answering, each choice conditioned on the query and the frames already chosen.
• Reward model (frozen): a reference video-LLM scores each selected frame set by how confidently it favors the correct answer, capturing its internal utility of that visual evidence.
• Reinforcement learning, no frame labels: the policy learns from these reward signals alone; frame-level supervision is never required, and semantically and temporally coherent frame compositions emerge implicitly.

Plug-and-Play Pipeline

Video (4 fps, up to 512 frames) + Question
  │
  ▼
[ReFoCUS Policy Model]     ← this model
  │
  │ 32 query-relevant frames (temporally sorted)
  ▼
[Downstream Video-LLM]     ← frozen, any off-the-shelf
  │
  ▼
Answer

Starting from the special <|startofframe|> token, the policy attends over the pool of frame embeddings and samples frames one at a time without replacement, so every choice is conditioned on the question and all prior selections. Its Mamba-2 (state-space) backbone keeps computation and memory linear in video length, and the whole selection runs in a single feed-forward autoregressive pass, with no iterative evaluation or post-processing.

Usage

The selected frame indices are temporally sorted and can be forwarded to any off-the-shelf video-LLM. For the full selection/inference pipeline and evaluation scripts, please refer to the ReFoCUS GitHub repository.

Citation

@InProceedings{Lee_2026_CVPR,
    author    = {Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man},
    title     = {ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings},
    month     = {June},
    year      = {2026},
    pages     = {8291-8302}
}

License

This model is released under the Apache 2.0 License.