Molmo2Fish
Molmo2Fish is Molmo2-8B fine-tuned to track fish in sonar video, and to edit those tracks in response to natural-language feedback.
Track correction is treated as a conversation: the model is shown a set of existing tracks — its own earlier predictions, another tracker's output, or corrupted ground truth — is told in words what is wrong with them, and returns a repaired set of tracks.
| Code | github.com/tidalove/molmo2fish |
| Data | tidalove/cfc-track-instruction |
| Source video | Caltech Fish Counting |
Files
| file | what it is |
|---|---|
*.safetensors, config.json, … | HuggingFace-format weights at the repo root — what vLLM and launch_scripts/hf_eval.py consume |
Molmo2Fish-step420-raw.tar | the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning |
Training
Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all three components — LLM, ViT, and connector — with the base weights frozen:
torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \
--lora_llm --lora_vit --lora_connector --lora_rank 64 \
--save_folder=/path/to/save/folder
The cfc_correction mixture combines pure tracking, targeted correction, synthetically
corrupted correction, correction of real model predictions at two quality levels
(molmo_high / molmo_low), and text-only correction. See the
dataset card for what each
config contains.
Citation
@article{molmo2fish,
title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance},
author={Kai van Brunt and Justin Kay and Sara Beery},
year={2026},
url={https://arxiv.org/abs/2608.18602}
}