Cohere finetune for Japanese focused on general/anime domains.
From v0.1, tuned hyperparameters improving CER across all test sets. Adjusted loss resulting in slightly higher insertion rate on certain audios but I think the insertions are nicer.
- GSPO with CE
- 2^17 steps
- 64 bsz
- Each sequence sample 1 greedy/7 temp1,topk20
- Audio length 0-35s, avg ~25.3s
- Frozen encoder, full finetune decoder
Bigger but faster than whisper-large-v3-turbo from more subsampling. Likely final checkpoint, will focus on finishing whisper training.
See my most recent whisper finetune for more evals
| tedx | jsut-book | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| cer | sr | ir | dr | cer | sr | ir | dr | ||
| cohere | 22.0 | 5.0 | 6.0 | 11.1 | 29.8 | 9.9 | 1.9 | 18.0 | |
| v0.1 | 14.4 | 4.7 | 7.5 | 2.2 | 18.3 | 11.2 | 0.9 | 6.3 | |
| ja | 12.9 | 4.8 | 6.0 | 2.1 | 17.5 | 10.9 | 0.8 | 5.8 |
cv/reazon removed due to contamination
| fleurs | jsut-basic | blue-archive | nekopara | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cer | sr | ir | dr | cer | sr | ir | dr | cer | sr | ir | dr | cer | sr | ir | dr | ||||
| cohere_b1 | 4.9 | 2.7 | 0.4 | 1.8 | 8.4 | 5.7 | 1.8 | 0.8 | 8.1 | 4.0 | 1.2 | 2.9 | 30.4 | 13.5 | 3.7 | 13.2 | |||
| cohere_b5 | 4.9 | 2.6 | 0.4 | 1.9 | 8.4 | 5.7 | 1.8 | 0.9 | 8.1 | 3.9 | 1.1 | 3.1 | 29.4 | 12.7 | 2.0 | 14.8 | |||
| v0.1_b1 | 5.2 | 2.8 | 0.4 | 1.9 | 6.9 | 5.0 | 1.1 | 0.8 | 6.8 | 3.9 | 1.4 | 1.5 | 24.8 | 14.0 | 3.1 | 7.7 | |||
| v0.1_b5 | 4.9 | 2.7 | 0.4 | 1.9 | 6.8 | 4.9 | 1.1 | 0.8 | 6.7 | 3.7 | 1.2 | 1.7 | 24.2 | 12.9 | 1.8 | 9.5 | |||
| ja_b1 | 5.0 | 2.8 | 0.5 | 1.7 | 6.7 | 4.9 | 1.0 | 0.8 | 6.6 | 3.8 | 1.5 | 1.4 | 24.4 | 14.1 | 3.5 | 6.8 | |||
| ja_b5 | 4.8 | 2.7 | 0.4 | 1.7 | 6.6 | 4.8 | 1.0 | 0.8 | 6.5 | 3.7 | 1.3 | 1.5 | 23.4 | 12.8 | 1.9 | 8.7 |
| air | clannad | himanatsu | kanon | sakuuta | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cer | sr | ir | dr | cer | sr | ir | dr | cer | sr | ir | dr | cer | sr | ir | dr | cer | sr | ir | dr | |||||
| cohere_b1 | 18.2 | 9.4 | 1.5 | 7.3 | 17.5 | 7.7 | 2.0 | 7.8 | 28.1 | 11.1 | 5.3 | 11.7 | 16.8 | 8.1 | 2.2 | 6.5 | 16.1 | 7.7 | 3.3 | 5.1 | ||||
| cohere_b5 | 18.1 | 9.2 | 1.4 | 7.5 | 17.6 | 7.5 | 1.8 | 8.2 | 25.6 | 10.1 | 2.4 | 13.1 | 16.7 | 7.9 | 2.0 | 6.8 | 15.5 | 7.3 | 2.4 | 5.8 | ||||
| v0.1_b1 | 16.0 | 9.3 | 1.6 | 5.0 | 12.5 | 6.8 | 1.2 | 4.4 | 20.6 | 12.2 | 3.5 | 4.9 | 14.2 | 8.0 | 1.6 | 4.5 | 10.9 | 6.7 | 2.2 | 2.1 | ||||
| v0.1_b5 | 15.8 | 9.3 | 1.5 | 5.0 | 12.4 | 6.7 | 1.1 | 4.6 | 20.3 | 11.9 | 2.9 | 5.5 | 14.2 | 8.1 | 1.5 | 4.6 | 10.8 | 6.5 | 2.0 | 2.2 | ||||
| ja_b1 | 15.8 | 9.2 | 1.6 | 5.0 | 12.3 | 6.7 | 1.2 | 4.4 | 20.5 | 12.1 | 3.5 | 4.9 | 13.9 | 7.9 | 1.6 | 4.4 | 10.8 | 6.6 | 2.2 | 2.1 | ||||
| ja_b5 | 15.4 | 9.0 | 1.3 | 5.2 | 12.1 | 6.5 | 0.9 | 4.7 | 19.7 | 11.3 | 2.3 | 6.0 | 13.9 | 7.9 | 1.3 | 4.7 | 10.6 | 6.3 | 1.9 | 2.3 |