SeloWhisper-ko-disfluency

Korean ASR with inline disfluency detection, fine-tuned from openai/whisper-large-v3-turbo.

Transcribes Korean speech while emitting 10 special tokens for fillers, repetitions, and laughter directly inside the transcript.

Note. Methodology, training data, and analysis are reserved for an upcoming paper. This repository releases only the model weights and the inference recipe.

Model Specification

Disfluency Tokens

Tokens are registered as additional_special_tokens. A dedicated <|pad|> token is introduced so EOS is preserved during label masking.

Usage

import torch, torchaudio
from transformers import WhisperProcessor, WhisperForConditionalGeneration

MODEL_ID = "rearleg/SeloWhisper-ko-disfluency"

processor = WhisperProcessor.from_pretrained(MODEL_ID)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_ID).eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

waveform, sr = torchaudio.load("sample.wav")
if sr != 16000:
    waveform = torchaudio.functional.resample(waveform, sr, 16000)
    sr = 16000
if waveform.shape[0] > 1:
    waveform = waveform.mean(dim=0, keepdim=True)

inputs = processor(
    waveform.squeeze().numpy(),
    sampling_rate=sr,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    generated = model.generate(
        inputs["input_features"],
        max_length=448,
        num_beams=1,
        do_sample=False,
    )

# Keep special tokens so disfluencies remain visible
transcription = processor.batch_decode(generated, skip_special_tokens=False)[0]
print(transcription)

Decode with skip_special_tokens=False to keep disfluency tags visible. Strip Whisper meta tokens (<|ko|>, <|transcribe|>, <|notimestamps|>, <|endoftext|>) manually if you only want the disfluency annotations.

This release ships a generation_config.json with forced_decoder_ids, suppress_tokens, and begin_suppress_tokens already cleared — no extra unsetting is required at inference time.

Performance (checkpoint-2500)

Evaluated on held-out Korean conversational speech.

Metric definitions

• CER — character error rate, whitespace included.
• WER — word error rate. Each disfluency token is treated as a single unit (substituted to a single private-use character during scoring) to avoid penalty inflation.
• sWER — WER after whitespace normalization, anchoring hypothesis spacing to the reference.
• sCER — CER on whitespace-stripped syllable streams (Korean-specific).
• Filler P / R / F1 — counter-based across the 10 disfluency tokens: TP = min(ref_count, hyp_count) per token, summed globally; FP / FN computed symmetrically.

Limitations

• Optimized for Korean spontaneous speech; not tuned for broadcast news, code-switching, or non-Korean speech.
• Like all Whisper-family models, hallucination is possible on silent or very short clips — a VAD or RMS-based silence filter upstream is recommended.

License

MIT — see LICENSE for the full text.

Citation

@misc{cheon2025selowhisper,
  title        = {SeloWhisper-ko-disfluency: Korean ASR with Inline Disfluency Detection},
  author       = {Cheon, Changhyun},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/rearleg/SeloWhisper-ko-disfluency}}
}

Whisper:

@article{radford2022whisper,
  title   = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author  = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg
             and McLeavey, Christine and Sutskever, Ilya},
  journal = {arXiv preprint arXiv:2212.04356},
  year    = {2022}
}