SpeechX
Multi-task speech classifier based on the Whisper-base encoder that predicts a speaker's gender and language from a short audio clip.
Use cases
- Fast language identification (LID) — 19 ms per clip end-to-end on a single GPU (well under 40–50 ms), or 270 clips/s batched; 109 languages, 97% top-5 accuracy
- Gender detection from voice — 97.8% accuracy, same single forward pass (both predictions come together)
- Pre-routing for ASR pipelines (pick the right transcription model per clip), call-center analytics, voice assistant profiling, dataset auditing/labeling
Model architecture
audio (16 kHz mono, short clips)
→ Whisper log-mel feature extractor (80 bins)
→ Whisper-base encoder (20.6M params, fp16)
→ mean pooling over time → 512-d embedding
→ Linear(512 → 2) gender head
→ Linear(512 → 109) language head
The Whisper decoder is not used. Weights are stored in fp16 (39 MB).
Training data
- Train: 423,772 clips (106 languages)
- Validation: 50,000 clips (108 languages, stratified)
Training: 3 epochs, AdamW, bf16, OneCycle LR (heads 1e-4, encoder 1e-5).
Results (validation, 50K clips)
Gender (2 classes)
| Metric | Value |
|---|---|
| Accuracy | 97.75% |
| Precision (macro) | 97.52% |
| Recall (macro) | 97.63% |
| F1 (macro) | 97.58% |
| Class | Precision | Recall | F1 |
|---|---|---|---|
| female | 0.967 | 0.972 | 0.969 |
| male | 0.984 | 0.981 | 0.982 |
Language (109 classes)
| Metric | Value |
|---|---|
| Top-1 accuracy | 87.16% |
| Top-1 accuracy (66 well-resourced langs, F1 ≥ 70%) | 89.96% |
| Top-5 accuracy | 97.10% |
| Precision (macro) | 72.68% |
| Recall (macro) | 67.77% |
| F1 (macro) | 68.66% |
Key languages (validation F1):
| Language | Precision | Recall | F1 |
|---|---|---|---|
| Arabic (ar) | 0.904 | 0.896 | 0.900 |
| English (en) | 0.869 | 0.859 | 0.864 |
| Spanish (es) | 0.834 | 0.848 | 0.841 |
| French (fr) | 0.892 | 0.829 | 0.859 |
| German (de) | 0.856 | 0.859 | 0.858 |
| Italian (it) | 0.816 | 0.873 | 0.844 |
| Portuguese (pt) | 0.735 | 0.862 | 0.794 |
| Russian (ru) | 0.783 | 0.830 | 0.806 |
| Turkish (tr) | 0.828 | 0.877 | 0.852 |
| Persian (fa) | 0.793 | 0.875 | 0.832 |
| Urdu (ur) | 0.821 | 0.858 | 0.839 |
| Hindi (hi) | 0.671 | 0.647 | 0.659 |
| Japanese (ja) | 0.955 | 0.943 | 0.949 |
| Korean (ko) | 0.933 | 0.949 | 0.941 |
| Chinese Mandarin (zh-CN) | 0.876 | 0.884 | 0.880 |
| Chinese Taiwan (zh-TW) | 0.800 | 0.823 | 0.811 |
| Indonesian (id) | 0.900 | 0.883 | 0.892 |
| Swahili (sw) | 0.918 | 0.938 | 0.928 |
| Polish (pl) | 0.814 | 0.836 | 0.825 |
| Ukrainian (uk) | 0.843 | 0.858 | 0.851 |
High-resource languages overall (ta, ka, hu, lg, rw…) reach 90–97% F1. Languages with < 50 training clips are unreliable.
Latency
| Device | Mode | Latency |
|---|---|---|
| NVIDIA L4 (fp16) | single clip (load + features + forward) | 18.6 ms (54 clips/s) |
| NVIDIA L4 (fp16) | batch 64 | 237 ms/batch → 270 clips/s |
| CPU (8 threads, fp32) | single clip | 317 ms (3.2 clips/s) |
GPU is fast enough for real-time LID/gender gating with a sub-50 ms budget per clip. CPU works for offline/batch processing; for real-time CPU use consider int8 quantization or distilling to whisper-tiny.
Usage
SpeechX is a native HF model (auto_map + trust_remote_code) — load it with AutoModel:
import torch
import torchaudio
from transformers import AutoModel, WhisperFeatureExtractor
model = AutoModel.from_pretrained(
"AhmedReda25300/SpeechX", trust_remote_code=True, dtype=torch.float16
).cuda().eval()
fe = WhisperFeatureExtractor.from_pretrained("AhmedReda25300/SpeechX")
wav, sr = torchaudio.load("clip.wav")
wav = wav.mean(dim=0) # mono
if sr != 16000:
wav = torchaudio.functional.resample(wav, sr, 16000)
x = fe(wav.numpy(), sampling_rate=16000, return_tensors="pt").input_features.cuda().half()
with torch.no_grad():
pred = model.predict(x) # convenience method
print(pred)
# {'gender': 'male', 'gender_confidence': 0.9571,
# 'language_topk': [{'lang': 'ar', 'prob': 0.312}, {'lang': 'fa', 'prob': 0.184}, ...]}
License
Apache-2.0. The Whisper base model is MIT-licensed (OpenAI); Common Voice audio is CC0.