Language of Motion — Decoder-only 4-part Audio-to-Motion (a2m)
Speech-driven, whole-body co-speech motion generation as next-token prediction. A pretrained LLM backbone (Qwen3) is turned into an audio→motion model by keeping its text vocabulary and enlarging it with GLM-4-Voice audio tokens and four motion codebooks, then generating the motion compositionally in a single causal stream:
[BOS] audio-tokens [SEP] [FACE] face [UPPER] upper [LOWER] lower [HAND] hand [EOS]
Each motion stream is decoded by its VQ decoder to SMPL-X (face = 6D head + 6D jaw + 100 expr; body = upper / lower / hand), giving a full talking + gesturing avatar from speech alone.
Checkpoints
| File | Backbone | Params | val loss ↓ | val acc ↑ |
|---|---|---|---|---|
qwen3-1.7B_4part.pt | Qwen3-1.7B | 1.76 B | 0.696 | 0.854 |
qwen3-0.6B_4part.pt | Qwen3-0.6B | 0.61 B | 0.705 | 0.834 |
(val loss = mean cross-entropy over the 4 motion streams on a held-out split; the two numbers are comparable to each other since both use the same vocab/targets.)
Each .pt is self-describing: {model, cfg, vocab, step, val_loss, val_acc} (torch.load).
Architecture
- Backbone: pretrained
Qwen3-1.7B/Qwen3-0.6B(decoder-only). The text vocab is kept; the embedding table is resized (mean-init) to append audio + motion + delimiter tokens. - Audio tokens: GLM-4-Voice (16384-way, 12.5 tok/s).
- Motion codebooks: ViBES face VQ (512-way, 25 tok/s, 112-D) + upper / lower / hand body VQs (256-way each, 6.25 tok/s).
- Generation: block-by-block (face → upper → lower → hand), each block constrained to its own code range; the decoded VQ streams are composited into a single SMPL-X sequence.
Training data
Trained on a mix of co-speech-gesture and talking-head corpora:
| Dataset | Provides | Notes |
|---|---|---|
| BEAT2 | body + face | CC BY-NC 4.0 (non-commercial) |
| TFHP (HDTF-derived) | face only (body masked) | talking-head |
| YouTube_Talking | face only (body masked) | in-the-wild talking-head |
For the YouTube_Talking data preparation, follow ViBES. We do not redistribute any raw
video/audio — YouTube_Talking is released recipe-only (URL list + downloader + derived
annotations) as part of ViBES:
JuzeZhang/YouTube_Talking — see its
README / the ViBES docs/1-data/youtube_talking.md for the download + annotation recipe. The audio
tokenization and motion-token extraction used here are the same scripts described there.
Usage / inference
These weights need the inference harness shipped alongside them (a2f_decoder/) plus the VQ
decoders and the GLM-4-Voice tokenizer. Minimal flow:
# 1. audio-tokens -> 4 motion streams -> SMPL-X
python a2f_decoder/infer_4part.py --ckpt qwen3-1.7B_4part.pt --out out/ --save_only
# 2. render (SMPL-X body via Blender + FLAME face)
See a2f_decoder/README.md (included) for the full dependency list (VQ checkpoints, SMPL-X / FLAME
model files, GLM-4-Voice tokenizer) and end-to-end example.
License
Research use only. These weights are derived from models trained on BEAT2 (CC BY-NC 4.0,
non-commercial), TFHP (HDTF-derived), and YouTube_Talking (source videos remain under YouTube's
terms; none are redistributed). The base models are Qwen3 (Apache-2.0). Use of these checkpoints is
restricted to non-commercial research and inherits the terms of the underlying datasets and
base models. See LICENSE.
Citation
@misc{lom_a2m_4part,
title = {Language of Motion: Decoder-only 4-part Audio-to-Motion},
author = {Zhang, Juze and others},
year = {2026},
note = {Research-use-only checkpoints, https://huggingface.co/JuzeZhang}
}