🌮 TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
Model description
We introduce TACO as a family of multi-modal large action models designed to improve performance on such complex, multi-step and multi-modal tasks. During inference, TACO produces chains-of-thought-and–action (CoTA), executes intermediate steps by invoking external tools such as OCR, depth estimation and calculator, then integrates both the thoughts and action outputs to produce coherent responses. Our TACO models outperform the instruction-tuned baseline across 8 benchmarks, achieving a 3.6% improvement on average, with gains up to 15% in MMVet tasks involving OCR, mathematical reasoning and spatial reasoning.
Usage
See our Github repository.
Intended uses & limitations
This model is intended to be used on complex, multi-step and multi-modal question answering tasks. It is trained to answer visual questions with some of the following 15 actions：OCR, LocalizeObjects, GetObjects,
EstimateRegionDepth, EstimateObjectDepth, Crop, ZoomIn, QueryLanguageModel, GetImageToImagesSimilarity, GetImageToTextsSimilarity,
GetTextToImagesSimilarity, DetectFaces, QueryKnowledgeBase, Calculate, and SolveMathEquation. Additionally, the Terminate action
is also supported for the model to provide a final answer.
For other types of tasks that don't benefit from the actions above, you might need to train a new model or further finetune it with other actions.
Training and evaluation data
See our paper for details.
Training procedure and hyperparameters
See our paper for details.
Training results
See our paper for details.
License information
This release is for research purposes only in support of an academic paper. This repository is licensed under the noncommercial license CC-BY-NC 4.0.
Citation
Please cite us if you find our repository helpful. Thank you!
@misc{ma2024tacolearningmultimodalaction,
title={TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action},
author={Zixian Ma and Jianguo Zhang and Zhiwei Liu and Jieyu Zhang and Juntao Tan and Manli Shu and Juan Carlos Niebles and Shelby Heinecke and Huan Wang and Caiming Xiong and Ranjay Krishna and Silvio Savarese},
year={2024},
eprint={2412.05479},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.05479},
}