Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

Unified Audio Schema is a novel holistic framework for audio supervision that disentangles and restructures supervision across transcription, paralinguistics, and non-linguistic events.

📄 Paper | 💻 GitHub

This repository provides our model checkpoints trained using Unified Audio Schema. For the complete codebase, please refer to the corresponding GitHub repository.

Model Details

Notes:

• The model supports interleaved text and audio input/output, enabling flexible multimodal interactions.
• Speech waveform reconstruction for generated audio tokens relies on the StableToken decoder.

Quick Start

Installation

git clone --recursive https://github.com/Tencent/Unified_Audio_Schema.git
cd Unified_Audio_Schema && pip install -r requirements.txt

Download Checkpoints

# Model weights
huggingface-cli download tencent/Unified_Audio_Schema --local-dir checkpoints/Unified_Audio_Schema

# StableToken decoder (required for speech waveform reconstruction)
huggingface-cli download tencent/StableToken --local-dir checkpoints/StableToken

Inference

import torch
import torchaudio
from src.model import UASAudio

model = UASAudio(
	model_path="checkpoints/Unified_Audio_Schema",
	audio_decoder_path="checkpoints/StableToken/decoder",
	device="cuda" if torch.cuda.is_available() else "cpu",
)

dialogue_system_prompt = (
	"User will provide you with a speech instruction. Do it step by step. "
	"First, think about the instruction and respond in a interleaved manner, "
	"with 13 text token followed by 52 audio tokens."
)

messages = [
	{"role": "system", "content": dialogue_system_prompt},
	{
		"role": "user",
		"content": [
			{"type": "audio", "audio": "assets/give_me_a_brief_introduction_to_the_great_wall.wav"},
		],
	},
	{"role": "assistant", "content": None},
]

generation_config = {
    "max_new_tokens": 4096,
    "temperature": 0.7,
    "repetition_penalty": 1.05,
    "top_p": 0.9,
    "do_sample": True
}

_, text, audio_tokens = model(messages, **generation_config)
print(text)

if len(audio_tokens) > 0:
	audio_array, sampling_rate = model.tokens_to_audio(audio_tokens)
	torchaudio.save("response.wav", audio_array, sampling_rate)

Supported Scenarios

Our model can be applied to a wide range of audio understanding and generation tasks, including:

• Text-input conversation
• Speech-input conversation
• Automatic Speech Recognition (ASR)
• Audio captioning
• Text-to-Speech (TTS)

For more runnable examples, please refer to example_usage.ipynb in the GitHub repository.

Evaluation Highlights

UAS-Audio demonstrates strong performance on audio understanding, ASR, and TTS benchmarks.

Audio Understanding

ASR & TTS

Citation

If you find Unified Audio Schema or our model useful for your research, please cite:

@misc{zhang2026transcriptionunifiedaudioschema,
	title={Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs}, 
	author={Linhao Zhang and Yuhan Song and Aiwei Liu and Chuhan Wu and Sijun Zhang and Wei Jia and Yuan Liu and Houfeng Wang and Xiao Zhou},
	year={2026},
	eprint={2604.12506},
	archivePrefix={arXiv},
	primaryClass={cs.CL},
	url={https://arxiv.org/abs/2604.12506},
}

@inproceedings{song2026stabletoken,
	title={StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient Speech{LLM}s},
	author={Yuhan Song and Linhao Zhang and Chuhan Wu and Aiwei Liu and Wei Jia and Houfeng Wang and Zhou Xiao},
	booktitle={The Fourteenth International Conference on Learning Representations},
	year={2026},
	url={https://openreview.net/forum?id=17DNmdQ9aU}
}

License

This project is licensed under the License Term of Unified_Audio_Schema.