Whisper Large-v3 Swiss German — commercially usable (MIT)
Swiss German speech to Standard German text. LoRA fine-tune of openai/whisper-large-v3,
merged into the weights, with a deliberately clean licence chain: Apache-2.0 base model, MIT
training data, MIT result.
This is the most accurate commercially usable checkpoint from our run series. For 2.4× the speed at 1.6 WER points less accuracy, see the turbo variant.
Most public Swiss German models inherit CC BY-NC from SwissDial and cannot be used commercially. This checkpoint avoids that: it is trained only on MIT-licensed FHNW/i4ds material.
Results
Measured on the FHNW/i4ds All Swiss German Dialects Test Set (MIT, 5750 clips, 12.7 h, 17 dialects), decoded with faster-whisper (CTranslate2, float16, beam 5) on an RTX 3080 Ti.
| Model | WER dev | WER final | CER | RTF |
|---|---|---|---|---|
| this checkpoint | 22.1 % | 24.7 % | 12.7 % | 0.080 |
openai/whisper-large-v3 (no fine-tune) | 24.0 % | 26.3 % | 13.7 % | 0.080 |
openai/whisper-large-v3-turbo (no fine-tune) | 26.5 % | 28.5 % | 14.6 % | 0.036 |
| our turbo fine-tune | 23.7 % | 25.9 % | 13.4 % | 0.033 |
1.9 WER points better than the untouched base model, at identical speed.
The test set is split speaker-disjointly into two halves: dev (2930 clips) was used for model selection, final (2820 clips) was held out. The final half is harder for every model measured, including untrained baselines — compare within a column, not across. The test set was never trained on.
WER/CER are per-clip means after normalisation (lowercase, punctuation removed, ß → ss).
Usage
faster-whisper / CTranslate2 — weights are in the ct2/ subfolder, so fetch that folder
first (the repository root holds the transformers format, whose config.json differs):
from huggingface_hub import snapshot_download
from faster_whisper import WhisperModel
path = snapshot_download("gcoli/whisper-large-v3-swiss-german-mit",
allow_patterns="ct2/*")
model = WhisperModel(f"{path}/ct2", device="cuda", compute_type="float16")
segments, info = model.transcribe("input.wav", language="de", beam_size=5)
print(" ".join(s.text.strip() for s in segments))
transformers — the repository root:
from transformers import pipeline
asr = pipeline("automatic-speech-recognition",
model="gcoli/whisper-large-v3-swiss-german-mit", device=0)
print(asr("input.wav", generate_kwargs={"language": "german", "task": "transcribe"})["text"])
Use language token de and task transcribe: the model outputs Standard German, not dialect
spelling. Audio should be 16 kHz mono.
Long recordings
On recordings of several minutes, turn context carry-over off. faster-whisper leaves
condition_on_previous_text on by default, and on 15-minute files it cost this model family
2.5 (turbo) to 11.7 (large) WER points in our measurements, with the output growing ~14 %
longer than the reference — the model starts inventing text. VAD filtering did not help either.
segments, info = model.transcribe("meeting.wav", language="de", beam_size=5,
condition_on_previous_text=False, vad_filter=False)
Measured on 26 files of ~15 min each, built by concatenating the held-out half of the test set, so the audio content is identical to the short-form numbers above. Expect roughly 1 to 3 WER points worse on long recordings than on single sentences; that gap differs per model and does not follow the short-form ranking.
Compatibility
- Format: safetensors (fp16) at the repository root, CTranslate2 float16 under
ct2/ - Runtimes: transformers, faster-whisper / CTranslate2
int8_float16quantisation costs about 0.3 WER points on this checkpoint's base family and is a reasonable trade for VRAM-constrained setups- For MLX (Apple Silicon), convert the root safetensors with
mlx_whisper
Training
| Base | openai/whisper-large-v3 (Apache-2.0) |
| Method | LoRA r=32, α=64, target q_proj/v_proj, dropout 0.05, merged after training |
| Optimiser | lr 3e-4, linear decay, 500 warmup steps, 1 epoch, batch 8 × 2 accumulation, bf16 + gradient checkpointing |
| Data | 20 h FHNW/i4ds Swiss Parliaments Corpus v2 (Bern) + 80 h Gemeinderat Zürich |
| Hardware | single RTX 3080 Ti (12 GB), ~2.2 h training |
The Zürich corpus is audio only. Transcripts were generated with openai/whisper-large-v3
(Apache-2.0) and filtered by confidence (avg_logprob > -0.45, no_speech_prob < 0.3,
compression ratio < 2.2, repetition-loop rejection).
Two findings from the run series this checkpoint comes from:
- Learning rate dominates. The same recipe at lr 1e-3 produced no gain at all over the base model; 3e-4 gained 1.8 points immediately.
- Source mixture matters more than volume. Both corpora must be present — Bern-only and
Zürich-only fine-tunes are clearly worse. On
large-v3a high Zürich share helps measurably (this checkpoint beats a 50/50 mix by 0.27 points, 95 % CI −0.50 to −0.04); onturbothe same ratio makes no difference. Doubling the Zürich pseudo-labels to 160 h changed nothing.
Other languages
Fine-tuning on one language variety can damage the others. Measured on google/fleurs
(250 validation clips per language, no prompt, beam 5):
| German | English | French | Italian | Spanish | |
|---|---|---|---|---|---|
| this checkpoint | 5.4 % | 5.1 % | 5.2 % | 2.9 % | 3.3 % |
openai/whisper-large-v3 | 4.3 % | 4.4 % | 4.4 % | 2.5 % | 3.1 % |
The fine-tune costs 0.2 to 1.1 WER points in the other languages — a real but mild regression, not catastrophic forgetting. If you need a multilingual system, keep the base model for everything except Swiss German.
Limitations
- The test set is read speech; the training data is parliamentary speech. Performance on spontaneous conversation, podcasts or noisy recordings is unmeasured.
- Training data covers two regions only (Bern, Zürich). Valais and Fribourg remain the weakest dialects at roughly 30–35 % WER.
- Zürich transcripts are pseudo-labels: the student cannot substantially exceed its teacher.
- Whisper can hallucinate or omit text, particularly with noise, silence, overlapping speakers, uncommon dialects or specialised vocabulary. Do not use its output as the sole basis for high-impact decisions. Obtain consent before transcribing people.
Provenance
- Base:
openai/whisper-large-v3 - Pseudo-label teacher:
openai/whisper-large-v3(Apache-2.0) - Training data: FHNW/i4ds Swiss Parliaments Corpus v2 and Gemeinderat Zürich Audio Corpus, both MIT, https://www.cs.technik.fhnw.ch/i4ds-datasets
- Evaluation: FHNW/i4ds All Swiss German Dialects Test Set (MIT), same source
- Details in
training_provenance.json
License and usage conditions
MIT. The base model (openai/whisper-large-v3) is Apache-2.0, the training data (FHNW/i4ds
Swiss Parliaments, Gemeinderat Zürich) is MIT, and the pseudo-labels were generated with the same
Apache-2.0 base model. Commercial use is permitted.
Please credit FHNW/i4ds for the underlying corpora. This repository grants no additional rights to upstream models, training data or software, and implies no endorsement by FHNW, i4ds or OpenAI.