🎙️ OmniBengali ASR — CTC 300M

Fine-Tuned Omnilingual ASR for Low-Resource Bengali Speech Recognition

Authors: Bhaskar Pal · Sagnik Basak

Key Results

Fine-tuning Meta's Omnilingual ASR CTC 300M on 50 hours of real-world noisy Bengali audio reduces WER from 65% to 47% and completely eliminates a critical baseline failure mode where the model incorrectly transcribes Bengali speech into Hindi, Tamil, and Telugu scripts.

The Problem We Solved

The baseline omniASR_CTC_300M model, despite being trained on 1600+ languages, has a fundamental limitation for Bengali: CTC models ignore the lang conditioning parameter. This means the model has no mechanism to know which language it is hearing, and in practice it frequently outputs the wrong script entirely for Bengali audio.

Bengali audio input: "আমি বাংলায় কথা বলছি"

Baseline output (wrong):    "मैं हिंदी में बात कर रहा हूं"  ← Hindi script
                         or  "நான் தமிழில் பேசுகிறேன்"      ← Tamil script  
                         or  "میں اردو میں بات کر رہا ہوں"  ← Urdu script

Fine-tuned output (correct): "আমি বাংলায় কথা বলছি"         ← Bengali ✓

Fine-tuning resolves this by baking Bengali phoneme-to-script associations directly into the encoder and CTC head weights — no runtime language conditioning needed.

Performance

Overall Metrics

Training Curve

The plot below shows CTC loss on train and validation splits over 10,000 steps. Loss dropped from ~215 → ~66 (train) and ~70 → ~59 (validation) with no divergence, indicating stable learning and good generalization.

Training curve image:

What the numbers mean

WER (Word Error Rate) — out of every 100 words spoken,
how many did the model get wrong?

Baseline:    65 words wrong out of 100
Fine-tuned:  47 words wrong out of 100
Improvement: 18 fewer errors per 100 words ✓

Additionally the baseline was outputting
the WRONG LANGUAGE entirely in most cases —
making the 65% WER misleading since it was
not even attempting Bengali.

Dataset

Training data is a private Bengali speech corpus and is not publicly released. The model weights only are shared under Apache 2.0.

Data characteristics:

• Source: Real-world Bengali audio (mixed domains, noisy conditions)
• Segmentation: ~10 second average clip length
• Ground truth: Derived from subtitle/transcript alignment
• Acoustic conditions: Mixed — indoor, outdoor, varied recording quality
• Script: Bengali (Unicode, ben_Beng)

Model Architecture

Input: Raw 16kHz mono audio waveform
         ↓
CNN Feature Extractor (wav2vec2 frontend)
  → Extracts local acoustic features every 20ms
         ↓
Wav2Vec2 Transformer Encoder (300M params)
  → 24 transformer layers, hidden dim 1024
  → Models long-range speech dependencies
         ↓
CTC Linear Projection Head
  → Maps 1024-dim features → 9812 token vocabulary
         ↓
CTC Decoding (greedy argmax + blank collapse)
  → Outputs Bengali Unicode text

Fine-tuning strategy:

• Encoder frozen for first 500 steps (CTC head only)
• Full model unfrozen from step 500 onwards
• This two-phase approach stabilizes early training before the full 300M parameters are updated

Usage

Installation

pip install omnilingual-asr fairseq2 torchaudio

Quick Start

import torch
from fairseq2.models.hub import load_model
from omnilingual_asr.models.inference.pipeline import (
    ASRInferencePipeline,
    load_tokenizer,
)
from huggingface_hub import hf_hub_download

# ── Load model ────────────────────────────────────────────────────────
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Download fine-tuned weights from HF Hub
model_path = hf_hub_download(
    repo_id="YOUR_USERNAME/omniASR-CTC-300M-bengali",
    filename="model.pt"
)

# Load base architecture
model = load_model(
    "omniASR_CTC_300M",
    device=device,
    dtype=torch.bfloat16
)

# Inject fine-tuned Bengali weights
state_dict = torch.load(model_path, map_location=device)
model.load_state_dict(state_dict, strict=False)

# Load tokenizer
tokenizer = load_tokenizer("omniASR_CTC_300M")

# Build inference pipeline
pipeline = ASRInferencePipeline(
    model_card=None,
    model=model,
    tokenizer=tokenizer,
    device=device,
    dtype=torch.bfloat16,
)

# ── Transcribe ────────────────────────────────────────────────────────
# From file path
result = pipeline.transcribe(["path/to/bengali_audio.wav"])
print(result[0])

# From multiple files (batched)
results = pipeline.transcribe(
    ["audio1.wav", "audio2.wav", "audio3.wav"],
    batch_size=4
)
for text in results:
    print(text)

Transcribe from microphone (real-time)

import sounddevice as sd
import numpy as np

SAMPLE_RATE = 16_000
DURATION    = 5  # seconds to record

print("Recording... speak now")
audio = sd.rec(
    int(DURATION * SAMPLE_RATE),
    samplerate=SAMPLE_RATE,
    channels=1,
    dtype=np.float32
)
sd.wait()
audio = audio.squeeze()

# Pass numpy array directly to pipeline
result = pipeline.transcribe([{"waveform": audio, "sample_rate": SAMPLE_RATE}])
print(f"Transcription: {result[0]}")

Batch transcription from a folder

from pathlib import Path

audio_dir = Path("path/to/your/bengali/audio/files")
wav_files  = list(audio_dir.glob("*.wav"))

print(f"Transcribing {len(wav_files)} files...")
results = pipeline.transcribe(
    [str(f) for f in wav_files],
    batch_size=8
)

for audio_file, text in zip(wav_files, results):
    print(f"{audio_file.name}: {text}")

Deploy

Option 1 — Hugging Face Inference API (easiest)

Once the model is public, you can call it directly via the HF Inference API:

import requests

API_URL = "https://api-inference.huggingface.co/models/YOUR_USERNAME/omniASR-CTC-300M-bengali"
headers = {"Authorization": "Bearer YOUR_HF_TOKEN"}

with open("audio.wav", "rb") as f:
    data = f.read()

response = requests.post(API_URL, headers=headers, data=data)
print(response.json())

Option 2 — Hugging Face Spaces (free demo)

Deploy a Gradio demo in minutes:

# app.py — paste this into a new HF Space
import gradio as gr
import torch
from fairseq2.models.hub import load_model
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline, load_tokenizer
from huggingface_hub import hf_hub_download

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_path = hf_hub_download(repo_id="YOUR_USERNAME/omniASR-CTC-300M-bengali", filename="model.pt")
model = load_model("omniASR_CTC_300M", device=device, dtype=torch.bfloat16)
state_dict = torch.load(model_path, map_location=device)
model.load_state_dict(state_dict, strict=False)
tokenizer = load_tokenizer("omniASR_CTC_300M")
pipeline = ASRInferencePipeline(model_card=None, model=model, tokenizer=tokenizer, device=device, dtype=torch.bfloat16)

def transcribe(audio):
    result = pipeline.transcribe([audio])
    return result[0]

demo = gr.Interface(
    fn=transcribe,
    inputs=gr.Audio(type="filepath", label="Upload Bengali Audio"),
    outputs=gr.Textbox(label="Bengali Transcription"),
    title="OmniBengali ASR — 300M CTC",
    description="Upload a Bengali audio file (.wav) to get Bengali text transcription.",
)
demo.launch()

Option 3 — Docker container

FROM python:3.10-slim
RUN pip install omnilingual-asr fairseq2 torchaudio fastapi uvicorn huggingface_hub
COPY serve.py .
CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000"]

# serve.py
from fastapi import FastAPI, UploadFile
import torch, tempfile, shutil
from pathlib import Path
from fairseq2.models.hub import load_model
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline, load_tokenizer
from huggingface_hub import hf_hub_download

app = FastAPI(title="OmniBengali ASR API")

device    = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_path = hf_hub_download(repo_id="YOUR_USERNAME/omniASR-CTC-300M-bengali", filename="model.pt")
model      = load_model("omniASR_CTC_300M", device=device, dtype=torch.bfloat16)
state_dict = torch.load(model_path, map_location=device)
model.load_state_dict(state_dict, strict=False)
tokenizer  = load_tokenizer("omniASR_CTC_300M")
pipeline   = ASRInferencePipeline(model_card=None, model=model, tokenizer=tokenizer, device=device, dtype=torch.bfloat16)

@app.post("/transcribe")
async def transcribe(file: UploadFile):
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
        shutil.copyfileobj(file.file, tmp)
        tmp_path = tmp.name
    result = pipeline.transcribe([tmp_path])
    Path(tmp_path).unlink()
    return {"transcription": result[0]}

⚠️ Limitations

• Trained on a specific Bengali speech domain
  Performance may vary on heavy dialects or accents
  not well represented in training data

• CTC decoding has no language model rescoring
  Output may contain disfluencies that an LM would clean up

• Optimised for ~10 second clips
  Very short (<1s) or very long (>30s) clips may degrade

• Training data is noisy real-world audio
  May perform differently on clean studio recordings

🔬 Technical Notes

Why fine-tuning fixes the cross-lingual confusion:

The CTC model has no runtime language conditioning — the lang parameter is documented as ignored for CTC inference. The baseline therefore outputs whatever script has the highest probability given the acoustic features alone, which for Bengali audio often means Hindi/Tamil/Telugu due to shared phonetic space in the pretrained multilingual representation. Fine-tuning on a single-language corpus biases both the encoder representations and the CTC projection head toward Bengali-specific phoneme-to-character mappings, eliminating the cross-lingual confusion without requiring any runtime language specification.

Bug fix contributed upstream:

During fine-tuning we identified and fixed a reproducible crash in wer_calculator.py where all-blank CTC predictions (common in early training steps) produced zero-length hypothesis sequences, causing BatchLayout.of() to raise ValueError: All lengths in seq_lens must be >= 1. The fix has been submitted as a PR to the upstream omnilingual-asr repository.

📜 Citation

If you use this model in your work, please cite:

@misc{omnibengali_asr_300m_2025,
  title     = {OmniBengali ASR — Fine-Tuned Omnilingual ASR CTC 300M for Bengali Speech Recognition},
  author    = {YOUR_FULL_NAME and COLLABORATOR_FULL_NAME},
  year      = {2025},
  url       = {https://huggingface.co/YOUR_USERNAME/omniASR-CTC-300M-bengali},
  note      = {Fine-tuned on ~50 hours of Bengali speech data}
}

Please also cite the original Omnilingual ASR work this model is based on:

@misc{omnilingualasr2025,
  title  = {Omnilingual ASR: Open-Source Multilingual Speech Recognition},
  author = {Meta AI Research},
  year   = {2025},
  url    = {https://github.com/facebookresearch/omnilingual-asr}
}

🤝 Acknowledgements

This model is a derivative of Meta AI's Omnilingual ASR project. We thank the Omnilingual ASR team for open-sourcing both the model weights and the fairseq2-based training recipe that made this fine-tuning possible.

Made with ❤️ for the Bengali NLP community

🤗 Model Hub · 🐛 Report Issue · 📄 Base Model