Ex-Omni is an public release for omni-modal response generation. Given text or speech input, the system can produce response text, speech units / decoded audio, and 52-dimensional facial blendshape coefficients, with optional rendering into a talking-face video.
📖 Table of Contents
🗂️ Repository Structure
.
├── asset/ # Download the mesh templates here, e.g. EmoTalk.npz
├── cosyvoice/ # Runtime audio decoder modules
├── deploy.py # Main Gradio entrypoint
├── deploy_base.py # Shared inference pipeline and UI logic
├── ex_omni/
│ ├── constants.py # Runtime constants
│ ├── flow_inference.py # Audio decoder wrapper
│ ├── render_utils.py # Blendshape rendering utilities
│ └── model/
│ ├── language_model/ # Omni model wrapper
│ ├── speech_encoder/ # Whisper speech encoder
│ ├── speech_projector/ # Speech projector
│ ├── speech_generator/ # Speech generator
│ └── blendshape_generator/ # Blendshape generator
├── requirements.txt # Python dependencies
└── LICENSE.txt # License file
⚙️ Quick Start
Installation
# 1. Create and activate environment
conda create -n Ex-Omni python=3.10 -y
conda activate Ex-Omni
# 2. Install PyTorch (example: CUDA 12.6)
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu1216
# 3. Install project dependencies
pip install -r requirements.txt
# 4. Install pytorch3d separately according to your CUDA / PyTorch version
# See official pytorch3d installation instructions
Checkpoints and Assets
Prepare the following assets before running inference:
| Component | Expected Path | Source |
|---|---|---|
| Ex-Omni checkpoint | ckpt/Ex-Omni/ |  |
| Flow decoder checkpoint | ckpt/glm-4-voice-decoder/flow.pt |  |
| HiFT decoder checkpoint | ckpt/glm-4-voice-decoder/hift.pt | |
| EmoTalk mesh template | asset/EmoTalk.npz |  |
| Claire mesh template | asset/claire.npz |  |
The Claire template is not redistributed in this repository. Please obtain the original Claire asset yourself and convert it into the
.npzformat.
Launch the Demo
python deploy.py \
--model-path ckpt/Ex-Omni \
--flow_ckpt_path ckpt/glm-4-voice-decoder/flow.pt \
--hift_ckpt_path ckpt/glm-4-voice-decoder/hift.pt \
--template_type emotalk \
--port 8080
Then open:
http://localhost:8080
🙏 Acknowledgements
We would like to thank the authors of OpenOmni, LLaMA-Omni2, EmoTalk and UniTalker. Parts of the implementation and overall system design were developed with reference to their open-source release.
📄 Citation
If you use this project, please cite our paper:
@misc{zhang2026exomnienabling3dfacial,
title={Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models},
author={Haoyu Zhang and Zhipeng Li and Yiwen Guo and Tianshu Yu},
year={2026},
eprint={2602.07106},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2602.07106},
}