TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation

TTA is a multilingual model that jointly supports transcribe, translate, and align tasks. It provides strong multilingual ASR/ST performance and cross-lingual speech retrieval capability.

🔗 Paper: https://arxiv.org/abs/2511.14410
🔗 Model: https://huggingface.co/AudenAI/auden-tta-m10
🔗 Encoder: https://huggingface.co/AudenAI/auden-encoder-tta-m10
🔗 Code: https://github.com/AudenAI/Auden/tree/main/examples/tta

🔍 What Can This Model Do?

• 🎙️ Multilingual ASR (transcribe)
• 🌍 Speech translation (translate)
• 🧩 Audio–text alignment (align)
• 🔎 Cross-lingual speech retrieval

Quick Start

TTA model

from auden.auto.auto_model import AutoModel

# 1) Load a model checkpoint directory (contains config.json + weights)
model_dir = "AudenAI/auden-tta-m10"  # or any exported directory / HF repo id
model = AutoModel.from_pretrained(model_dir)
model = model.to("cuda")
model.eval()

# 2) Prepare input features (x, x_lens). If you have raw audio, you can use
#    model.speech_encoder.extract_feature(wav) to get (x, x_lens).
x, x_lens = ...  # Tensor shapes: (B, T, F), (B,)

inputs = (x, x_lens)
# Alternatively, you can pass WAV inputs directly:
# - List of WAV paths (str):
#   inputs = ["/abs/a.wav", "/abs/b.wav"]
# - List of mono waveforms (Tensor/ndarray), 16 kHz:
#   inputs = [torch.randn(16000*5), torch.randn(16000*3)]

# 3a) Transcribe (RNNT greedy)
out = model.generate(inputs, task="transcribe", blank_penalty=0.0, return_timestamps=False)
print(out["hypotheses"])  # list[str]

# 3b) Translate (attention beam search). Language can be a single str or a list[str] per utterance
out = model.generate(
    inputs,
    task="translate",
    beam_size=5,
    source_language=["zh"] * x.size(0),
    target_language=["en"] * x.size(0),
)
print(out["hypotheses"])      # list[str]
print(out["source_language"]) # list[str], model-predicted or provided
print(out["target_language"]) # list[str], model-predicted or provided

# 3c) Align (audio-text similarity)
texts = ["hello world", "good morning"]
out = model.generate(inputs, task="align", texts=texts)
print(out["similarities"])  # (B, len(texts))
print(out["audio_emb"]) # (B, emb_dim)
print(out["text_emb"]) # (B, emb_dim)

TTA encoder

from auden.auto.auto_model import AutoModel
encoder = AutoModel.from_pretrained("AudenAI/auden-encoder-tta-m10")
encoder = encoder.to("cuda")

# 2) Prepare input features (x, x_lens). If you have raw audio, you can use
#    encoder.extract_feature(wav) to get (x, x_lens).
x, x_lens = ...  # Tensor shapes: (B, T, F), (B,)

encoder_output = encoder(x, x_lens)
print(encoder_output["encoder_out"]) # (B, T//4, D)
print(encoder_output["encoder_out_lens"]) # (B)

📌 Model Characteristics

• Input: Raw audio waveform (16 kHz recommended)
• Output: Transcription, translation, or alignment scores
• Encoder: TTA encoder (AudenAI/auden-encoder-tta-m10)
• Tasks: transcribe / translate / align

📊 Evaluation

Multilingual ASR & ST

TTA Encoder (LLM-ASR Encoder Evaluation)

Training Data

Full data composition (open-source links + in-house aggregation):

Language totals from the same table:

⚠️ Limitations

• Performance depends on audio quality and recording conditions.
• For long-form audio, chunking and post-processing might be required for optimal performance.
• Not designed for safety-critical applications.

Citation

If you use this model in your research, please cite:

@article{liu2025tta,
  title={TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation},
  author={Liu, Wei and Li, Jiahong and Shao, Yiwen and Yu, Dong},
  journal={arXiv preprint arXiv:2511.14410},
  year={2025}
}