TinyOctopus: Bilingual Audio Language Model 🐙🔊
📢 Overview
TinyOctopus is a Bilingual Audio Language Model (Audio-LLM) designed to process and generate text from audio inputs. The model leverages Distil-Whisper (distil-large-v3) for audio encoding, a cross-attention projection layer for alignment, and DeepSeek 1.5B for text generation. TinyOctopus is optimized for tasks such as:
- Bilingual Automatic Speech Recognition (ASR) 🗣️
- Arabic to English Speech Translation 🌍
- Spoken Arabic Dialect Identification
TinyOctopus maintaining the architectural principles of the following structure:
🏗 Model Architecture
TinyOctopus integrates:
- Distil-Whisper (distil-large-v3) for encoding audio inputs.
- Cross-Attention Projection Layer (trainable) to align audio features with textual representations.
- DeepSeek 1.5B as the core language model for text generation.
📂 Dataset
The model has been trained on multiple datasets to optimize its performance across different tasks:
-
QASR Dataset: QASR is the largest transcribed Arabic speech corpus, collected from the broadcast domain. It contains 2,000 hours of multi-dialect speech sampled at 16kHz from Al Jazeera News Channel, with lightly supervised transcriptions aligned with the audio segments. Unlike previous datasets, QASR includes linguistically motivated segmentation, punctuation, speaker information, and more. The dataset is suitable for ASR, Arabic dialect identification, punctuation restoration, speaker identification, and NLP applications. Additionally, a 130M-word language model dataset is available to aid language modeling. Speech recognition models trained on QASR achieve competitive WER compared to the MGB-2 corpus, and it has been used for downstream tasks like Named Entity Recognition (NER) and punctuation restoration.
-
ADI17 Dataset: ADI17 is a large-scale Arabic Dialect Identification (DID) dataset, collected from YouTube videos across 17 Arabic-speaking countries in the Middle East and North Africa. It contains 3,000 hours of speech for training DID systems and an additional 57 hours for development and testing. The dataset is categorized into short (<5s), medium (5-20s), and long (>20s) speech segments for detailed evaluation. ADI17 enables state-of-the-art dialect identification and provides a robust evaluation platform. It has been benchmarked on domain-mismatched conditions using the Multi-Genre Broadcast 3 (MGB-3) test set.
⚙️ Installation & Usage
💻 Install Dependencies
pip install -r requirements.txt
Inference
from inference import transcribe
audio_path = "path/to/audio.wav" # Replace with your actual audio file
output = transcribe(audio_path, task="asr") # Options: "dialect", "asr", "translation"
print("Generated Text:", output)
How to Try It?
You can test the model by uploading or recording your own audio files using the Gradio demo:
➡️ Try the Model
Evaluation Results
ASR Performance (WER & Error Breakdown)
| Tasks | WER (%) | Substitution (%) | Deletion (%) | Insertion (%) |
|---|---|---|---|---|
| ASR_QASR (Arabic) | 16.00 | 9.5 | 2.7 | 3.8 |
| ASR_ibrispeech&tedlium (English) | 4.50 | 3.0 | 0.8 | 0.7 |
Translation Performance (BLEU Scores)
| Tasks | BLEU (GPT-4o) | BLEU (Google) |
|---|---|---|
| Translation | 55.05 | 43.23 |
Dialect Identification Accuracy
| Tasks | Accuracy (%) |
|---|---|
| Dialect Identification | 70.59 |
Examples
Example 1: Arabic Speech Recognition
🎵 Audio Input (Arabic):
📝 User Prompt:
Transcribe the audio or قم بتفريغ المقطع الصوتي
💡 System Response:
أهلا بكم مشاهدينا الكرام في حلقة جديدة من برنامج الاقتصاد والناس
🎵 Audio Input (English):
📝 User Prompt:
Transcribe the audio or قم بتفريغ المقطع الصوتي
💡 System Response:
NO IT'S NOT TOO SOON
Example 2: Arabic to English Translation
🎵 Audio Input:
📝 User Prompt:
Translate the following Arabic speech into English or قم بترجمة المقطع للإنجليزية
💡 System Response:
I took a loan a certain amount of money to pay off the debt
Example 3: Dialect Identification
🎵 Audio Input:
📝 User Prompt:
Identify the dialect of the given speech or ماهي لهجة المتحدث؟
💡 System Response:
KSA