Malayalam dialect identification (5 Kerala districts): XLSR + logistic regression

What it does

Give it a recording of someone speaking Malayalam, and it predicts which of 5 Kerala districts the speaker comes from. It uses how they speak (accent, pronunciation, prosody), not what they say.

A second classifier covers the 4 districts without Kollam.

This is a research baseline for Malayalam dialect identification, trained on the companion dataset Vaani Malayalam Dialect-ID (derived from VAANI; see Citation).

Input: audio files of Malayalam speech.

• Any format librosa reads, at any sample rate; audio is resampled to 16 kHz mono.
• Use at least 2 s of speech. Several clips from one speaker work best.

Output: one JSON line per file:

{"file": "clip.wav", "windows": 3, "prediction": "Thiruvananthapuram",
 "probabilities": {"Kasaragod": 0.0009, "Kozhikode": 0.0062, "Wayanad": 0.0003,
                   "Kollam": 0.0281, "Thiruvananthapuram": 0.9646}}

With --speaker, all files are pooled as one speaker into an extra combined prediction. That is more reliable: 63–68% per speaker, against 52–55% per clip.

Usage

pip install torch transformers librosa numpy
python3 predict.py my_clip.wav                              # 5 districts
python3 predict.py my_clip.wav --task 4class_noKollam       # 4 districts
python3 predict.py a1.wav a2.wav a3.wav --speaker           # several clips of one speaker

Everything needed is included: the XLSR encoder in xlsr/, and the classifier as plain NumPy arrays (classifier_*.npz). No scikit-learn and no internet download are needed. A GPU is used if available; the CPU also works.

How it works

1. The audio is cut into 2 s windows with a 1 s hop.
2. Each window goes through the XLSR encoder in xlsr/. This is a wav2vec2-XLSR model (facebook/wav2vec2-large-xlsr-53, 24 layers, 1024-d) fine-tuned for Malayalam ASR. Hidden states of transformer layer 9 are averaged over time.
3. The vector is standardised and passed to a multinomial logistic regression: softmax(W·z + b).
4. The window probabilities are averaged per clip, and optionally per speaker.

Why layer 9. All 25 layers were tried, and the layer was chosen on dev accuracy. Accuracy peaks in the middle layers (8–12) and drops in the top layers, which ASR fine-tuning trained to ignore accent.

Results (test speakers never seen in training)

5-class test confusion (clips; rows = true district)

Reading the errors:

• Errors are mostly between geographic neighbours: Kasaragod ↔ Kozhikode in the north, Kollam ↔ Thiruvananthapuram in the south.
• Kollam was recorded in a separate collection batch, yet its recall is not inflated.

Every per-layer number is in RESULTS.md and results.json.

Training

The classifiers in this repository were trained in the public Kaggle notebook Malayalam Dialect-ID: Training, on the Kaggle copy of the dataset. The notebook downloads the frozen encoder, embeds every clip, trains and selects the per-layer classifiers, and writes these exact files. Re-running it reproduces the numbers below.

Training data

• Dataset: Vaani Malayalam Dialect-ID, 350 speakers (35 F + 35 M per district) × 30 clips, about 14 h. It is a subset of VAANI (CC-BY-4.0).
• Content control: only clips from generic images shown in all districts are used, so the district can't be read from what is described.
• Splits: speaker-disjoint, 42 / 14 / 14 speakers per district for train / dev / test.
• Tuning: the layer and the regularisation strength C were chosen on dev. C = 0.01 for 5-class and 0.001 for 4-class.

Limitations

• Wayanad (recall 0.79–0.85) probably benefits partly from its recording conditions, not only dialect.
• Small test set (14 speakers per district): confidence intervals are about ±11 points.
• The label is the speaker's district of residence, not a linguist's judgement.
• One speaking style: phone-recorded picture description. Accuracy on films, broadcast or conversation is unknown and probably lower.
• Only 5 of Kerala's 14 districts are covered. Speakers from other districts are always forced into one of the 5.
• This is a frozen-encoder baseline. Self-supervised encoders or full fine-tuning will likely do better.

Ethical use: research only. Do not use it to identify, profile or discriminate against people by accent or origin.

Files

Reproducing

python3 scripts/build_vaani_dialect.py --src <folder with the VAANI Kerala shards>   # or download the Kaggle dataset
python3 scripts/01_embed_xlsr.py --data <dataset root>      # window embeddings, all 25 layers
python3 scripts/02_train_eval.py --tag xlsr_ml_asr          # per-layer classifiers, dev-selected, test report

Or simply re-run the training notebook on Kaggle. All random seeds are fixed.

Citation

The training audio comes from VAANI. Please cite it:

@misc{pulikodan2026vaanicapturinglanguagelandscape,
  title={VAANI: Capturing the language landscape for an inclusive digital India},
  author={Sujith Pulikodan and Abhayjeet Singh and Agneedh Basu and Nihar Desai and Pavan Kumar J and Pranav D Bhat and Raghu Dharmaraju and Ritika Gupta and Sathvik Udupa and Saurabh Kumar and Sumit Sharma and Vaibhav Vishwakarma and Visruth Sanka and Dinesh Tewari and Harsh Dhand and Amrita Kamat and Sukhwinder Singh and Shikhar Vashishth and Partha Talukdar and Raj Acharya and Prasanta Kumar Ghosh},
  year={2026},
  eprint={2603.28714},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2603.28714}
}

The encoder is based on facebook/wav2vec2-large-xlsr-53 (Apache-2.0).