Whisper Telephony AMD (Answering Machine Detection)

A real-time audio classifier that detects whether a telephony call is answered by a human, voicemail, IVR system, or answering machine — using Whisper's speech understanding to distinguish human-recorded voicemail greetings from live speech.

Results

98.75% accuracy on 400 test samples with only 5 misclassifications:

                   precision    recall  f1-score   support

            human       1.00      0.99      1.00       114
        voicemail       0.96      0.99      0.98       102
              ivr       1.00      0.99      0.99        92
answering_machine       0.99      0.98      0.98        92

         accuracy                           0.99       400
        macro avg       0.99      0.99      0.99       400
     weighted avg       0.99      0.99      0.99       400

Confusion Matrix (rows = actual, columns = predicted):

Accuracy Per Epoch

Early stopping triggered after epoch 6 (patience=5, best at epoch 4). Best model loaded from epoch 4 checkpoint.

Model Details

Quick Start

Pipeline (simplest)

from transformers import pipeline

classifier = pipeline("audio-classification", model="AbijahKaj/whisper-telephony-amd")
result = classifier("phone_call.wav")
print(result)
# [{'score': 0.98, 'label': 'human'}, {'score': 0.01, 'label': 'voicemail'}, ...]

Manual Inference

from transformers import WhisperForAudioClassification, AutoFeatureExtractor
import torch

model = WhisperForAudioClassification.from_pretrained("AbijahKaj/whisper-telephony-amd")
fe = AutoFeatureExtractor.from_pretrained("AbijahKaj/whisper-telephony-amd")

# audio_array: numpy array at 16kHz
inputs = fe(audio_array, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    pred = torch.argmax(logits, dim=-1).item()
    label = model.config.id2label[str(pred)]
    print(f"Predicted: {label}")

Streaming Real-Time Inference

from streaming_amd import StreamingAMDClassifier

classifier = StreamingAMDClassifier("AbijahKaj/whisper-telephony-amd")

for pcm_chunk in audio_stream:  # 160ms chunks @ 8kHz
    result = classifier.process_chunk(pcm_chunk)
    if result:
        label, confidence, elapsed_s = result
        print(f"{label} ({confidence:.0%}) after {elapsed_s:.1f}s")
        break

Why Whisper?

Voicemail greetings are recorded by real humans — they are acoustically identical to live speech. Traditional acoustic-only models (energy, pitch, spectral features) cannot reliably distinguish "Hi, I'm not available, leave a message" from "Hello? Who's calling?".

Whisper's encoder was pre-trained on 680K hours of speech and understands what is being said, not just how it sounds. This semantic understanding is critical for AMD.

Training

Dataset

AbijahKaj/telephony-amd-dataset — 8,264 train / 400 test samples, balanced across 4 classes (~2,000 each).

Data sources:

Hyperparameters

Framework Versions

• Transformers 5.7.0
• PyTorch 2.11.0+cu130
• Datasets 4.8.5
• Tokenizers 0.22.2

Classes

Limitations

• Trained primarily on English, French, Spanish, and German audio
• TTS-generated non-human classes may not fully represent all real-world telephony systems
• Best performance on first 10 seconds of audio
• Not tested on noisy cellular connections or VoIP codec artifacts beyond telephony bandpass (300-3400Hz)
• The model may confuse voicemail greetings with answering machine messages in edge cases (2 misclassifications in test set)

Files

• model.safetensors — Model weights (31.7MB)
• config.json — Model configuration
• preprocessor_config.json — Feature extractor config
• streaming_amd.py — Streaming real-time inference module
• train_local.py — Training script (CLI args, RTX 5090 ready)

Citation

@misc{whisper-telephony-amd,
  author = {AbijahKaj},
  title = {Whisper Telephony AMD: Real-Time Answering Machine Detection},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/AbijahKaj/whisper-telephony-amd}
}