DiCoW_v3_MLC — BUT-FIT Model for MLC-SLM Challenge

This repository contains the DiCoW_v3_MLC model developed by BUT Speech@FIT for the MLC-SLM Challenge. Diarization-Conditioned Whisper (DiCoW) is a novel approach to target-speaker ASR that leverages speaker diarization outputs as conditioning information.

This model is available under the terms of CC BY 4.0. It incorporates an MIT-licensed base model and CC BY 4.0 licensed training data.

The model is described in detail in the following papers:

• 📰 Journal paper (main DiCoW paper): DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition
• 📰 ICASSP paper (initial DiCoW experiments): Target Speaker ASR with Whisper
• 📰 MLC-SLM Challenge submission paper: BUT System for the MLC-SLM Challenge

Model Summary

The model is based on Whisper large-v3-turbo, initially trained on:

• NOTSOFAR-1
• AMI Meeting Corpus
• Libri2Mix dataset

It is then fine-tuned on the MLC-SLM dataset as part of the MLC-SLM Challenge.

Model Details

• Developed by: BUT Speech@FIT, Brno University of Technology
• Model type: Whisper large-v3-turbo + DiCoW composition
• Language(s): Multilingual (primarily English, but supports multiple languages)
• License: apache-2.0
• Fine-tuned from: openai/whisper-large-v3-turbo
• Challenge: MLC-SLM (Multilingual Conversational Speech Language Model)

Model Sources

• Training Code: TS-ASR-Whisper GitHub
• Inference Code & DiCoW framework: DiCoW GitHub

Getting Started

from transformers import AutoModelForSpeechSeq2Seq

MODEL_NAME = "BUT-FIT/DiCoW_v3_MLC"
dicow = AutoModelForSpeechSeq2Seq.from_pretrained(MODEL_NAME, trust_remote_code=True)

For detailed inference and full pipelines, refer to: 👉 DiCoW GitHub inference repo

tcpWER/CER (%) on the MLC-SLM development set

Results marked with an asterisk () are reported using tcpCER, following the official evaluation protocol.*

Notes:

• GT = Ground-Truth Segmentation
• Real diar = Real Diarization
• Baseline uses Whisper large-v3 with chunked inference + finetunned Pyannote diarization.
• DiCoW uses fine-tuned DiariZen diarization.

Citation

If you use this model, please cite:

@article{POLOK2026101841,
    title = {DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition},
    journal = {Computer Speech & Language},
    volume = {95},
    pages = {101841},
    year = {2026},
    issn = {0885-2308},
    doi = {https://doi.org/10.1016/j.csl.2025.101841},
    url = {https://www.sciencedirect.com/science/article/pii/S088523082500066X},
    author = {Alexander Polok and Dominik Klement and Martin Kocour and Jiangyu Han and Federico Landini and Bolaji Yusuf and Matthew Wiesner and Sanjeev Khudanpur and Jan Černocký and Lukáš Burget},
    keywords = {Diarization-conditioned Whisper, Target-speaker ASR, Speaker diarization, Long-form ASR, Whisper adaptation},
}

@INPROCEEDINGS{10887683,
    author={Polok, Alexander and Klement, Dominik and Wiesner, Matthew and Khudanpur, Sanjeev and Černocký, Jan and Burget, Lukáš},
    booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
    title={Target Speaker ASR with Whisper}, 
    year={2025},
    volume={},
    number={},
    pages={1-5},
    keywords={Transforms;Signal processing;Transformers;Acoustics;Speech processing;target-speaker ASR;diarization conditioning;multi-speaker ASR;Whisper},
    doi={10.1109/ICASSP49660.2025.10887683}
}

@misc{polok2025mlcslmchallenge,
    title={BUT System for the MLC-SLM Challenge}, 
    author={Alexander Polok and Jiangyu Han and Dominik Klement and Samuele Cornell and Jan Černocký and Lukáš Burget},
    year={2025},
    eprint={2506.13414},
    archivePrefix={arXiv},
    primaryClass={eess.AS},
    url={https://arxiv.org/abs/2506.13414}, 
}

Contact

For questions or collaborations, feel free to email: ipoloka@fit.vut.cz
BUT Speech@FIT, Brno University of Technology
GitHub: BUTSpeechFIT