AudioX: Multilingual Speech-to-Text Model

AudioX is a state-of-the-art Indic multilingual automatic speech recognition (ASR) model family developed by Jivi AI. It comprises two specialized variants—AudioX-North and AudioX-South—each optimized for a distinct set of Indian languages to ensure better accuracy. AudioX-North supports Hindi, Gujarati, and Marathi, while AudioX-South covers Tamil, Telugu, Kannada, and Malayalam. Trained on a combination of open-source ASR datasets and proprietary audio, the AudioX models offer robust transcription capabilities across accents and acoustic conditions, delivering industry-leading performance across supported languages.

Purpose-Built for Indian Languages:

AudioX is designed to handle diverse Indian language inputs, supporting real-world applications such as voice assistants, transcription tools, customer service automation, and multilingual content creation. It provides high accuracy across regional accents and varying audio qualities.

Training Process:

AudioX is fine-tuned using supervised learning on top of an open-source speech recognition backbone. The training pipeline incorporates domain adaptation, language balancing, and noise augmentation for robustness across real-world scenarios.

Data Preparation:

The model is trained on:

• Open-source multilingual ASR corpora
• Proprietary Indian language medical datasets,

This hybrid approach boosts the model’s generalization across dialects and acoustic conditions.

Benchmarks:

AudioX achieves top performance across multiple Indian languages, outperforming both open and commercial ASR models. We evaluated AudioX on the Vistaar Benchmark using the official evaluation script provided by AI4Bharat’s Vistaar suite, ensuring rigorous, standardized comparison across diverse language scenarios.

🔧 Try This Model

You can easily run inference using the 🤗 transformers and librosa libraries. Here's a minimal example to get started:

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model and processor
device = "cuda"
processor = WhisperProcessor.from_pretrained("jiviai/audioX-south-v1")
model = WhisperForConditionalGeneration.from_pretrained("jiviai/audioX-south-v1").to(device)
model.config.forced_decoder_ids = None

# Load and preprocess audio
audio_path = "sample.wav"
audio_np, sr = librosa.load(audio_path, sr=None)
if sr != 16000:
    audio_np = librosa.resample(audio_np, orig_sr=sr, target_sr=16000)

input_features = processor(audio_np, sampling_rate=16000, return_tensors="pt").to(device).input_features

# Generate predictions
# Use ISO 639-1 language codes: "hi", "mr", "gu" for North; "ta", "te", "kn", "ml" for South
# Or omit the language argument for automatic language detection
predicted_ids = model.generate(input_features, task="transcribe", language="ta")

# Decode predictions
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)