VTInstructor-8B

VTInstructor-8B is the final SoTA checkpoint of VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments (ACM MM 2026).

This release is the VP-GRPO model: Qwen3-VL-8B-Instruct + VP-Adapter, after supervised fine-tuning and then VP-GRPO refinement, which updates the VP-Adapter gates together with the language decoder while the vision tower and the VP encoder stay frozen. It generates natural-language navigation instructions from egocentric RGB trajectories in continuous environments (R2R-CE / RxR-CE).

• Paper: arXiv:2608.15284
• Code: github.com/TidalHarley/VTInstructor
• Backbone: Qwen/Qwen3-VL-8B-Instruct

What this model does

Given a first-person trajectory (RGB keyframes + discrete actions), VTInstructor writes a concise indoor navigation instruction that matches the full path. Visual Trajectory Prompts (path ribbon, turn arrows, goal marker) and a 3-channel semantic mask are injected through a lightweight VP-Adapter; the same checkpoint also works on plain RGB renders without VTP, because the trajectory context is already absorbed into the weights.

Typical uses:

1. Instruction generation on R2R-CE / RxR-CE trajectories
2. Data augmentation for downstream VLN followers (see §3 below)
3. Drop-in instruction replacement on VLN-CE .json.gz splits via the open-source generation script

This model is intended for non-commercial research on vision-and-language navigation. See License.

Results

All numbers below are copied from the ACM MM 2026 paper. Figures are the original tables.

1. Navigation instruction generation (NLG)

VTInstructor is the best model on both R2R-CE Val Unseen and RxR-CE Val Unseen, including against GPT-5.4, Gemini-3.1-Pro-Preview, and much larger open-source VLMs. Relative to the strongest baseline it gains +0.357 CIDEr on R2R-CE and +0.109 CIDEr on RxR-CE.

R2R-CE Val Unseen

RxR-CE Val Unseen

Comparison with prior speaker methods trained on discretized R2R / RxR viewpoint graphs (those numbers are not on the same continuous-env protocol):

2. Frozen VLN follower

Instructions from VTInstructor are executed by a frozen CorrectNav follower on R2R-CE Val Unseen. VTInstructor improves frozen-follower success by 14.7 percentage points over the strongest baseline, and matches or exceeds official human instructions on SR / OSR / NE.

3. Data augmentation for a VLN follower

Training CorrectNav (LLaVA-Video-7B backbone) with human R2R-CE+RxR-CE data plus VTInstructor-generated instructions yields about +3 SR on both Val Unseen splits.

R2R-CE Val Unseen

RxR-CE Val Unseen

Files in this repo

Both vp_encoder.pt and vp_adapters.pt ship in this repo, so pointing CKPT / EVAL_CKPT at the downloaded directory is enough — no extra files are needed.

transformers==4.57.6 is the validated version. Newer transformers may break the VP-Adapter injection into Qwen3-VL internals.

Loading only with Qwen3VLForConditionalGeneration.from_pretrained gives the language/vision backbone; to reproduce paper numbers you must attach the VP modules from the code repo.

Quick start

# 1. weights
hf download TidalYang/VTInstructor-8b --local-dir ./VTInstructor-8b

# 2. code
git clone https://github.com/TidalHarley/VTInstructor.git && cd VTInstructor
pip install -r requirements.txt

Generate instructions (RGB-only, or with VTP masks if the split has them):

CKPT=../VTInstructor-8b \
DATA_DIR=/path/to/rendered_split \
DATASET_TYPE=r2rce \
OUT_JSON=outputs/augment/r2r_style.json \
  bash generate/run_generate.sh

DATASET_TYPE=rxrce switches to the longer RxR-style narration. VP_MODE=auto (default) uses vp_masks when present and falls back to RGB otherwise. MODEL_DIR is optional — the processor files in this repo are used by default. GEN_CUDA selects GPUs (default 0); shards are merged automatically.

To reproduce the paper's benchmark numbers instead (needs rendered val_unseen splits):

EVAL_CKPT=../VTInstructor-8b \
MODEL_DIR=/path/to/Qwen3-VL-8B-Instruct \
EVAL_CUDA=0 \
  bash eval/run_eval.sh

Set EVAL_CUDA to the GPU ordinals you actually have — it defaults to all eight. Scoring additionally needs pycocoevalcap and a Java runtime on PATH (for METEOR / SPICE).

Training recipe (how this checkpoint was produced)

1. SFT of Qwen3-VL-8B-Instruct + VP-Adapter on filtered R2R-CE / RxR-CE visual trajectories
2. VP-GRPO RL with a weighted NLG reward (BLEU-1/4, METEOR, ROUGE-L, CIDEr), updating the VP-Adapter gates and the language decoder while the vision tower and the VP encoder stay frozen → this checkpoint

See the GitHub README for rendering, VTP mask construction, SFT, GRPO, and evaluation.

Citation

If you use this model, please cite:

@misc{yang2026vtinstructorvisualtrajectoryprompting,
      title={VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments},
      author={Haolin Yang and Yuxing Long and Zihan Yang and Hao Dong},
      year={2026},
      eprint={2608.15284},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.15284},
}

License

The code release is Apache 2.0, and these weights are distributed under the same terms.

This model was trained on data derived from R2R / R2R-CE, RxR / RxR-CE and Matterport3D, which are distributed under CC BY-NC-SA 3.0 US and the Matterport3D Terms of Use. Please respect those terms — the checkpoint is intended for non-commercial research use. The Qwen3-VL-8B-Instruct backbone license also applies.