full_moonshine_tiny_all_data_wup1_ep3_lr1e-4
Full fine-tune of UsefulSensors/moonshine-tiny (27.1M parameters) for Macedonian speech-to-text, trained on the whole corpus: the vezilka-asri collection plus seven curated sets -- videa_so_transkript_od_yt, alfa_audios, sitel_audios, doniraj (accepted donations), mozzila_common_voice, fleurs_mk and jargon -- ~2.99M clips of at most 30 s. The dialect repos and the held-out test set are deliberately left out.
This is the best recipe of the earlier warmup/LR sweep (full_moonshine_tiny_wup1_ep3_lr1e-4, rachno WER 14.24) with the data changed and nothing else -- same 3 epochs, same learning rate, schedule, warmup, batch size and eval cadence -- so the gain is attributable to the curated sets that were added.
Moonshine is a raw-waveform encoder-decoder: it takes input_values rather than
Whisper's fixed 30 s log-mel window, so its cost scales with actual audio length
instead of being flat per clip. That is what makes it viable on-device, and it is
why its real-time factor barely moves with utterance length where Whisper's does.
Evaluation
Held-out test set rachno_provereno_od_yt (44 clips, 0.32 h),
cuda, batch 16, 1 beam.
| metric | value |
|---|---|
| WER | 11.62 |
| CER | 4.21 |
| SER | 97.73 |
| Parameters | 27.1 M |
| Latency | 72 ms/utterance |
| Throughput | 366x real-time |
| RTF | 0.00273 |
SER is exact-match over whole utterances. These test clips average 62 words, so at this WER an exactly-correct utterance is vanishingly unlikely -- one of the 44 lands exactly right after normalization -- read WER/CER.
Best WER seen during training (on the same held-out set): 11.33 at step 251000.
Compared with the other Moonshine runs
The two all-data runs
Compared with Whisper
The full table across every model evaluated is in comparison_all.md.
Training
| setting | value |
|---|---|
| learning rate | 0.0001 |
| warmup ratio | 0.01 |
| epochs | 3 |
| batch size | 32 x 1 accum |
| scheduler | cosine |
| weight decay | 0.01 |
| precision | bf16 |
| total steps | 280137 |
Usage
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
import librosa, torch
model_id = "h-gajdov/full_moonshine_tiny_all_data_wup1_ep3_lr1e-4"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id).eval()
wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
feats = processor.feature_extractor([wav], sampling_rate=16000,
return_tensors="pt", padding=True)
with torch.no_grad():
ids = model.generate(**feats, max_length=192)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
max_length is capped at 192 because the decoder has 194 positions
(max_position_embeddings). Moonshine has no forced language/task tokens, so do
not pass language= or task= to generate() -- it will raise.