Foresight — Qwen3-VL-2B SFT
The supervised fine-tuned vision-language model behind Foresight, a navigation policy that iteratively discovers instruction-relevant visual clues and refines its motion plans for open-world navigation.
- Project page: https://amrl.cs.utexas.edu/foresight/
- Paper: https://amrl.cs.utexas.edu/foresight/static/pdfs/ForesightPreprint.pdf
What this checkpoint does
Given a natural-language goal and a short history of RGB observations, the model proposes an image-space trajectory and critiques its own proposal. Motion planning and critique were co-trained into this single checkpoint, so both roles are served by one vLLM engine at deployment time.
Converting the accepted plan into metric BEV waypoints is handled by a separate grounding policy, which is not part of this repository.
Usage
The deployment stack loads this model through vLLM by repo ID. In
legged_deployment/config/vllm_server_sft.yaml:
model_overrides:
vlm:
provider_kwargs:
model: "ut-amrl/foresight-qwen3vl-2b-sft"
It also loads directly with transformers:
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
model = Qwen3VLForConditionalGeneration.from_pretrained(
"ut-amrl/foresight-qwen3vl-2b-sft", dtype="bfloat16", device_map="auto"
)
processor = AutoProcessor.from_pretrained("ut-amrl/foresight-qwen3vl-2b-sft")
Prompts are constructed by the Foresight codebase; see https://github.com/ut-amrl/foresight_public for the runnable deployment stack.
Training
Fine-tuned from Qwen/Qwen3-VL-2B-Instruct with LoRA (rank 64, alpha 64) applied to
all linear projections in both the language and vision towers. The adapter has been
merged into the base weights, so no PEFT dependency is needed at inference.
Weights are bfloat16.
Citation
@article{zhang2026foresight,
title={Foresight: Iterative Reasoning About Clues that Matter for Navigation},
author={Zhang, Arthur and Qi, Carl and Su, Donne and Meng, Xiangyun and Zhang, Amy and Biswas, Joydeep},
journal={arXiv preprint arXiv:2606.12550},
year={2026}
}