nb-whisper-dialect-id-4dialect-monotonize
A NbAiLab/nb-whisper-medium encoder fine-tuned for 4-class Norwegian dialect identification on F0-monotonized speech.
Given an audio clip of F0-monotonized Norwegian speech, the model predicts which of four broad dialect regions the speaker belongs to:
| Label | Region |
|---|---|
east | Eastern Norwegian (østnorsk) |
west | Western Norwegian (vestnorsk) |
mid | Central/Trøndersk Norwegian (trøndersk) |
north | Northern Norwegian (nordnorsk) |
This model is one of three "prosody-condition" models (unmodified / low-pass / monotonize) trained for:
Phoebe Parsons, Heming Strømholt Bremnes, Knut Kvale, Torbjørn Svendsen, and Giampiero Salvi. (2025). Effects of Prosodic Information on Dialect Classification Using Whisper Features. In Proceedings of Interspeech 2025, pages 2785–2789. doi: 10.21437/Interspeech.2025-200
The other two conditions:
scribe-project/nb-whisper-dialect-id-4dialect— unmodified audioscribe-project/nb-whisper-dialect-id-4dialect-low-pass— low-pass filtered audio
See the did_prosody_whisper repo for the training/evaluation code and the audio manipulation scripts used for the paper.
Model description
The model is NbAiLab/nb-whisper-medium's encoder with a classification head on top
(WhisperForAudioClassification), fully fine-tuned (no layers frozen) for sequence classification
over 4 dialect labels. Unlike the unmodified-audio model, this model was trained and evaluated on
speech whose F0 (pitch) contour was monotonized — flattened to the utterance's average F0. This
removes pitch-based prosodic variation while keeping segmental/phonetic content intact, isolating
the contribution of non-pitch cues (timing, energy, phonetic content) to dialect classification.
Intended uses & limitations
Intended for research on Norwegian dialect identification, specifically on studying how much
dialect information survives when pitch-based prosody is removed. Audio fed to this model should
be F0-monotonized the same way as during training (see the
did_prosody_whisper repo) — running
unmodified audio through this model does not reproduce the paper's "unmodified" condition results;
use scribe-project/nb-whisper-dialect-id-4dialect for that. It is not intended for consequential
decisions about individuals (e.g. hiring, legal, or identity-verification contexts).
Known limitations:
- Trained only on adult parliamentary speech; accuracy on other domains (conversational speech, children, non-native speakers) is not guaranteed.
- This repo ships the checkpoint from the final training epoch (epoch 3), for consistency with the unmodified and low-pass models in this series, rather than the epoch-2 checkpoint, which had marginally higher eval accuracy (0.8733 vs 0.8727 — see Training results below).
- The 4 dialect categories are coarse regional groupings, not a fine-grained dialect taxonomy, and boundaries between adjacent regions are inherently fuzzy.
How to use
import torch
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
import librosa
model_id = "scribe-project/nb-whisper-dialect-id-4dialect-monotonize"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModelForAudioClassification.from_pretrained(model_id)
# audio should be F0-monotonized the same way as during training — see the
# did_prosody_whisper repo for the monotonization procedure
audio, sr = librosa.load("path/to/monotonized_audio.wav", sr=feature_extractor.sampling_rate, mono=True)
inputs = feature_extractor(
audio,
sampling_rate=feature_extractor.sampling_rate,
return_tensors="pt",
)
with torch.no_grad():
logits = model(**inputs).logits
predicted_id = torch.argmax(logits, dim=-1).item()
print(model.config.id2label[predicted_id])
Audio should be mono, resampled to 16kHz. Clips longer than 30 seconds were randomly subsampled to 30 seconds during training.
Training and evaluation data
Trained on the SSC (Storting/Parliament Speech Corpus), using a speaker-disjoint train/validation split (no speaker overlap between train and eval). Each example is a single-speaker audio segment labeled with one of the 4 dialect categories above. Prior to training, each utterance's F0 contour was monotonized to its average F0 (estimated with REAPER); see the did_prosody_whisper repo for the exact Praat scripts used.
Training procedure
Training hyperparameters
- learning_rate: 3e-05
- train_batch_size: 16 (per device)
- eval_batch_size: 32 (per device)
- seed: 0
- distributed_type: multi-GPU
- num_devices: 6
- gradient_accumulation_steps: 2
- total_train_batch_size: 192
- optimizer: Adam (betas=(0.9, 0.999), epsilon=1e-08)
- lr_scheduler_type: linear, warmup_ratio 0.1
- num_epochs: 3.0
- mixed_precision_training: Native AMP
- feature encoder: not frozen (full fine-tune)
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 0.028 | 0.9989 | 452 | 0.7168 | 0.8503 |
| 0.0153 | 2.0 | 905 | 0.5684 | 0.8733 |
| 0.0084 | 2.9967 | 1356 | 0.6343 | 0.8727 |
The weights published in this repo are from epoch 3 (checkpoint-1356), the final training checkpoint — not the epoch-2 checkpoint, which had marginally higher eval accuracy but is inconsistent with the checkpoint selection used for the unmodified and low-pass conditions above.
Framework versions
- Transformers 4.40.0
- PyTorch 2.2.2+cu121
- Datasets 2.19.0
- Tokenizers 0.19.1