whisper-he-100m
A 94.4M-parameter Hebrew speech recognition model. It reaches 12.99% word error rate on ivrit.ai's eval-d1, which is 2.6 times fewer parameters than whisper-small for 13.8 points better Hebrew.
It is the larger sibling of itayinbar/whisper-base-he: same Hebrew tokenizer, same 3,112.7 hours of training data, twice the depth.
Results
Scored with a harness that reproduces the ivrit.ai Hebrew leaderboard on 40 of 40 published model and dataset pairs.
| Benchmark | this model, 94M | whisper-base-he, 50M | whisper-small, 242M | ivrit.ai, 809M |
|---|---|---|---|---|
ivrit-ai/eval-d1 | 12.99% | 14.87% | 5.5% | |
imvladikon/hebrew_speech_kan | 16.62% | 18.47% | 37.42% | 8.10% |
ivrit-ai/eval-whatsapp | 21.67% | 25.77% | 6.1% | |
google/fleurs he | 32.19% | 33.74% | 45.94% | 18.72% |
How it was built
Whisper offers no checkpoint near 100M parameters: base is 72.6M and small is
241.7M. This closes that gap by depth up-scaling, repeating each pretrained
layer of whisper-base in place, so the network begins as a strictly deeper
version of a model that already works rather than from noise. Layers are
interleaved (0, 0, 1, 1, 2, 2, ...) rather than concatenated, which keeps each
copy next to the representation it was trained to expect.
That gives 12 encoder and 12 decoder layers, 94.36M parameters after the Hebrew tokenizer graft, and it improved every benchmark over the 50.2M model by 1.5 to 4.1 points.
Whisper's 51,865-token multilingual vocabulary is replaced with an 8,192-token Hebrew byte-level BPE. On held-out Hebrew speech Whisper spends 3.17 tokens per word against this vocabulary's 1.76. Every new embedding row is initialised from the pretrained table rather than from noise.
Usage
from transformers import AutoModelForSpeechSeq2Seq, AutoTokenizer, AutoFeatureExtractor
from transformers.generation.utils import GenerationMixin
model = AutoModelForSpeechSeq2Seq.from_pretrained("itayinbar/whisper-he-100m")
tokenizer = AutoTokenizer.from_pretrained("itayinbar/whisper-he-100m")
features = AutoFeatureExtractor.from_pretrained("itayinbar/whisper-he-100m")
inputs = features(audio_16khz, sampling_rate=16000,
return_tensors="pt", padding="max_length")
# Monolingual: no language token, no task token, no timestamps.
ids = GenerationMixin.generate(model, **inputs, max_new_tokens=200)
print(tokenizer.decode(ids[0], skip_special_tokens=True))
Three things differ from stock Whisper:
- Call the generic
generate. There are no language or task tokens. - Pad mel features to the full 30-second window.
- Cut audio longer than 30 seconds into overlapping windows and stitch them.
A browser downloads 299.3 MB, the fp16 encoder with the fp32 decoder. Only those weights are shipped: an fp16 decoder fails to load in ONNX Runtime and an int8 encoder loads but changes the transcript. Every combination offered here was checked by transcribing with it and comparing against the fp32 output.
If size matters more than the last two points of accuracy, use whisper-base-he at 160 MB.
Training
3,112.7 hours over 552,327 utterances, all human-transcribed: 2,756.6 hours of
ivrit-ai/knesset-plenums, 295.0 of ivrit-ai/crowd-transcribe-v5, 44.7 of
ivrit-ai/crowd-recital, 9.5 of google/fleurs he and 6.9 of
imvladikon/hebrew_speech_kan. Schedule-free AdamW, learning rate 1e-4,
effective batch 16, bf16, 69,032 steps on a single RTX 5070 Laptop GPU.
Licence and provenance
Weights are Apache-2.0, following openai/whisper-base.
Training data from ivrit.ai under the
ivrit.ai licence, which permits training
models including commercially and requires attribution. FLEURS is CC-BY-4.0.
imvladikon/hebrew_speech_kan declares no licence on the Hub; it contributed 6.9
of 3,112.7 hours and is named so anyone relying on provenance can judge it.
Code and method: github.com/itayinbarr/Hebrew-small-asr