w2v-bert-2.0-music-lyrics-ctc
A character-level CTC lyrics model for full music mixes, i.e. vocals with the band, no source separation. It's fine-tuned from w2v-bert-2.0-music, with the encoder unfrozen. It is multilingual: the 921-token vocabulary covers Latin, Cyrillic, Greek, Arabic, Hebrew, Thai, kana/kanji and Hangul (decomposed into jamo).
Uses:
- Lyrics transcription (greedy CTC decoding).
- Forced alignment of known lyrics to audio (CTC Viterbi), which gives word and syllable onsets.
- Frame-level vocal activity and vocal F0/voicing from the auxiliary heads.
import torch
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor
processor = Wav2Vec2BertProcessor.from_pretrained("mrfakename/w2v-bert-2.0-music-lyrics-ctc")
model = Wav2Vec2BertForCTC.from_pretrained("mrfakename/w2v-bert-2.0-music-lyrics-ctc").eval()
# 16 kHz mono; up to ~30 s per call (see "Long audio" below)
inputs = processor(audio_16khz_mono, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits # (1, frames at 50 Hz, 921)
text = processor.batch_decode(logits.argmax(-1))[0]
Output format
- Lowercase, NFKC-normalised text with punctuation removed. Apostrophes are kept inside words.
- Hangul comes out as conjoining jamo. Use
unicodedata.normalize("NFC", text)to recompose syllables. - Token 0 is the CTC blank, and
|is the word delimiter / filler. - The filler also absorbs vocals that aren't lyrics (ad-libs, humming). For alignment, put
|between words and at both ends of the target sequence. - Section and performance tags (
[Chorus],(backing vocals)) are not transcribed.
Long audio
The model was trained on windows of up to 30 s. For longer audio, run overlapping windows (e.g. 30 s with 2 s of context on each side), keep the centre of each window, and concatenate the logits before decoding or aligning.
Auxiliary heads
aux_heads.safetensors holds two linear heads that operate on the final hidden states
(output_hidden_states=True, last layer):
vad.{weight,bias}: 1 logit per frame, vocal activity.f0.{weight,bias}: 361 outputs per frame. Index 0 is the voicing logit; indices 1..360 are vocal pitch bins of 20 cents each (bin b →10 * 2 ** ((20 * b + 1997.38) / 1200)Hz, ≈32 Hz–2 kHz).
Training
Fine-tuned from w2v-bert-2.0-music with weak supervision from whole-song lyrics:
- Full-song CTC bootstrap.
- Forced alignment and line-based chunking; the worst-aligned 15% of chunks are dropped.
- CTC on ≤ 30 s chunks, plus random windows cut anywhere in a song, so the model doesn't rely on inputs starting at a line boundary.
The vocal-activity and F0/voicing heads were trained jointly as auxiliary losses (weight 0.1 each).
Evaluation
Held-out songs, greedy decoding, no language model:
| CER | |
|---|---|
| Line-aligned chunks | 12.8% |
| Arbitrary windows (not cut at line boundaries) | 13.3% |
The reference lyrics are imperfect: the model often recovers words the references miss, so the true error is somewhat lower.
Forced-alignment timing: a word/syllable onset evaluation will be added here.
- CTC emissions are peaky, so onsets from forced alignment are usable, but word offsets (ends) are not reliable.
- For boundaries within about 20 ms, a dedicated boundary model is needed on top of this encoder.