Qwen2.5-Omni-7B SFT (with observations) — OmniGAIA fine-tune

This is a fully fine-tuned version of Qwen/Qwen2.5-Omni-7B on the OmniGAIA training data, trained as a tool-integrated omni-modal agent that natively reasons over video + audio + image and calls external tools (web_search, page_browser, code_executor, read_image/audio/video).

The key differences from the recipe in the OmniGAIA paper are:

• Training data source. We use the raw with_obs trajectories from the OmniGAIA train split as-is, rather than re-synthesizing trajectories via hindsight-guided tree exploration (the paper's OmniAtlas-SFT recipe).
• Long-context handling via sliding window. Long trajectories are split via sliding window (width=8192, overlap=4096) at user/assistant pair boundaries, so all observation context still fits in the model context window during training instead of being truncated.
• Role-level masking equivalent to the paper's masked SFT. Original role=tool (observation) messages are normalized to role=user to fit the Qwen2.5-Omni chat template. Combined with ms-swift's default --loss_scale default (which only computes loss on assistant turns), this means only assistant tokens contribute to the loss; tool / observation content is fully masked, functionally the same as the paper's trajectory-level masked SFT.

The training objective is functionally equivalent to OmniAtlas-SFT (only assistant tokens are supervised; tool / observation tokens are masked). The improvement over the paper's 7B numbers is therefore attributable to the training-data recipe (raw with_obs trajectories + sliding-window long-context handling), not to a different loss target.

This is checkpoint-150 (epoch ≈ 1.28), which gave the most stable validation performance during training.

Evaluation on OmniGAIA (Pass@1, official protocol)

Evaluated using the official OmniGAIA eval script (run_base_agent_original_paper.py) with DeepSeek-V3.2 as the LLM judge, same tools/active-perception setup as published baselines.

Comparison to OmniGAIA paper baselines (Qwen2.5-Omni-7B family)

The cross-seed variance comes mainly from (i) real-time web_search results drifting between runs, and (ii) DeepSeek-V3.2 judge non-determinism, not from the model itself (vLLM is run in greedy mode).

Per-difficulty breakdown (averaged over 4 seeds)

Training

This checkpoint corresponds to global step 150 / 234 (epoch ≈ 1.28). Later checkpoints (200, 234) showed slightly higher train loss reduction but no monotonic Pass@1 gain in our evaluation, so we use step 150 for the leaderboard submission.

Data contamination self-check

We verified that no test-set question text from test_metadata.json appears verbatim in any training sample (0 / 2,151 overlap). The training set shares media with the test set (~23%) because the OmniGAIA dataset is constructed from a shared media pool — this is the same condition under which all OmniAtlas / Qwen baselines in the paper were evaluated, so the comparison is apples-to-apples.

How to use

from transformers import Qwen2_5OmniForConditionalGeneration, AutoProcessor

model_path = "ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only"

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_path,
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True,
)

For best results with OmniGAIA-style multi-turn tool-use evaluation, serve it with vLLM:

vllm serve <model_path> \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 32768

and drive it via the OmniGAIA agent loop (run_base_agent_original_paper.py --enable-active-perception ...) with web_search, page_browser, code_executor, and read_image/audio/video tools.

Intended use & limitations

• Designed for omni-modal agentic QA with multi-turn tool use over video + audio + image.
• Inherits all limitations of Qwen2.5-Omni-7B (audio perception errors on long clips, vision recognition errors on rare entities, etc.).
• The training data is heavy on tool-call trajectories — direct one-shot answers without tool use are not what this checkpoint is optimized for.

Caveats and known limitations (read this before comparing to the paper)

1. Functionally equivalent loss masking, different role wrapper. tool / observation messages are remapped to role=user (to satisfy the Qwen2.5-Omni chat template), and ms-swift then masks all non-assistant tokens via --loss_scale default. The end effect — supervising only assistant tokens (reasoning + tool_call) — matches the paper's trajectory-level masked SFT, but the on-disk role schema differs. Downstream tooling that special-cases role=tool may not recognize the observation segments in this dataset / format.

2. Trajectories are not re-synthesized. The paper's OmniAtlas-SFT uses hindsight-guided tree exploration with Gemini-3-Flash to synthesize new tool-integrated trajectories. This model trains directly on the raw with_obs trajectories shipped in the OmniGAIA train split, without any re-synthesis or trajectory cleanup.

3. No DPO / RLHF stage. The paper reports its strongest 7B number with OmniAtlas-SFT + OmniDPO. This model is SFT-only.

4. Shared media pool between train and test. The OmniGAIA train and test sets are constructed from a shared media pool (~23% of train trajectories share at least one media file with a test item, even though no test-set question text appears verbatim in any train sample — see "Data contamination self-check" above). This is a property of the benchmark itself, the same as all baselines in the paper.

What this implies for the reported numbers:

• The 15.76 Pass@1 figure is achieved under the same evaluation protocol and the same effective training objective (masked SFT) as the paper's 7B baselines. The headroom over the paper's +OmniAtlas-SFT (11.4) and +OmniDPO (13.3) therefore reflects differences in training data recipe (raw with_obs + sliding window) rather than a different loss mask.
• A clean apples-to-apples ablation (paper-style trajectory synthesis vs raw with_obs, holding everything else fixed) has not been run in this codebase, so the per-component contribution of "raw with_obs" vs "sliding window" vs "different trajectory pool" is not yet decomposed.
• Use this checkpoint as a strong reproducible 7B starting point for OmniGAIA. If you intend to extend this for a publication, please run the corresponding ablations.

License

Apache-2.0, inherited from the base Qwen2.5-Omni-7B.

Citation

If you use this model, please also cite the underlying base model and the OmniGAIA benchmark:

@misc{qwen25omni,
  title  = {Qwen2.5-Omni Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

@misc{omnigaia,
  title  = {OmniGAIA: Towards Native Omni-Modal AI Agents},
  author = {RUC-NLPIR et al.},
  year   = {2026},
  url    = {https://huggingface.co/spaces/RUC-NLPIR/OmniGAIA-Leaderboard},
}