wav2vec2-xls-r-300m-korean-english
A jamo + Latin CTC acoustic model fine-tuned from facebook/wav2vec2-xls-r-300m
for Korean / English code-switched speech. It emits per-frame character
probabilities over a compact jamo + Latin vocabulary.
Training
- Base:
facebook/wav2vec2-xls-r-300m(300M multilingual wav2vec2), feature encoder frozen. - Head: fresh CTC head over a ~105-token jamo + Latin vocab.
- Data: 17.4 h of clean TTS-synthesized KO/EN/mixed speech (~13.5k clips, 0.3–15 s), roughly 57% code-switched / 22% English / 21% Korean.
- Recipe: 15 epochs, lr 3e-4, batch 16, bf16, best checkpoint by CER.
Important: jamo preprocessing
The vocab is jamo (decomposed Hangul) + Latin, not raw syllables. You MUST decompose target text the same way, and recompose model output:
import unicodedata
def to_jamo(t): # text -> model target space (NFD on Hangul only)
return "".join(unicodedata.normalize("NFD", c) if "가" <= c <= "힣" else c for c in t)
def from_jamo(t): # model output -> readable text
return unicodedata.normalize("NFC", t)
Usage
import soundfile as sf, unicodedata
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
proc = Wav2Vec2Processor.from_pretrained("raniczan/wav2vec2-xls-r-300m-korean-english")
model = Wav2Vec2ForCTC.from_pretrained("raniczan/wav2vec2-xls-r-300m-korean-english").eval()
wav, sr = sf.read("clip.wav") # resample to 16 kHz first if needed
x = proc(wav, sampling_rate=16000, return_tensors="pt").input_values
ids = model(x).logits.argmax(-1)[0]
print(unicodedata.normalize("NFC", proc.decode(ids)))