saga-2

By Capacit A/S. CC BY-NC 4.0, attribution required when shared. See License.

Danish automatic speech recognition. Ranked #1 on the Danish ASR Leaderboard.

thorhojhus/whisper-large-v3-turbo-danish's encoder paired with a Qwen3-0.6B-Base decoder through a frame-stacking projector. 1.24 B parameters, bf16. Output is cased and punctuated Danish.

Try out saga-2 here

Usage

pip install -r requirements.txt

from saga2 import load

model = load("capacit-ai/saga-2-m")          # CUDA if available, else CPU
print(model.transcribe("meeting.mp3"))
print(model.transcribe(["a.wav", "b.flac"], batch_size=16))

From the command line:

python run.py clip.wav                                 # weights from this directory
python run.py --repo capacit-ai/saga-2-m a.wav b.mp3   # or fetch from the hub
python run.py --json out.jsonl *.flac                  # one {"file","text"} line per file

Any sample rate or container librosa can read. Audio longer than 30 s is split at silences into pieces of at most 28 s and rejoined automatically.

Results

Measured with the leaderboard's own evaluation harness and normaliser (NFKC, spoken-number canonicalisation, hesitation fillers removed from reference and hypothesis alike), greedy decoding, on the five public Danish test sets. Percentages, lower is better.

268x real time on one RTX 5090 at batch 16.

Checkpoint selection used an in-training validation mix that includes a slice of the FLEURS da_dk test split, so the FLEURS column is mildly optimistic. The other four columns are untouched by selection.

Why these weights changed

An earlier release of saga-2 used an encoder we had fine-tuned ourselves from stock openai/whisper-large-v3-turbo. Swapping in thorhojhus/whisper-large-v3-turbo-danish as the encoder and continuing training scored better on four of the five test sets and on the mean, so those are the weights published here. There is no reason to ship the weaker model when the better one exists, and the repository name and usage stay the same so existing code keeps working.

Architecture

16 kHz audio -> log-mel (128 bins)
  -> Whisper-large-v3-turbo encoder                   32 layers, d_model 1280, 20 heads
  -> frame stack x4                                   one token per 80 ms, 375 per 30 s
  -> RMSNorm(5120) + Linear(5120 -> 1024)             projector, 5.25 M params
  -> Qwen3-0.6B-Base decoder                          28 layers, hidden 1024, 16 heads / 8 KV
  -> cased, punctuated Danish

The encoder keeps Whisper's absolute sinusoidal position embeddings, frozen throughout. The position table is sliced to the actual window rather than requiring a padded 30 s block, so short clips cost proportionally less compute.

The decoder sees audio embeddings spliced into its token stream as [audio][<|im_start|>] text [<|im_end|>], and generation stops on <|im_end|> (151645), not the tokenizer's <|endoftext|>. The bundled saga2 package handles this; calling generate directly requires setting eos_token_id=151645.

Whisper turbo's 4-layer decoder loops on short conversational clips. Replacing it with a full 28-layer Qwen3-0.6B language model is the main quality change in saga-2.

Training data

Training datasets used on the final run, no more than 2 epochs were used of each.

Augmentation

Channel augmentation on 60 % of rows (room impulse response, MUSAN noise, or codec simulation) at 0 to 20 dB SNR, VTLP on 35 %, speed perturbation, gain ±10 dB, SpecAugment. Trailing non-speech tails on 25 % of clips and 2.5 % of rows replaced with non-speech carrying an empty transcript, which is why the model returns an empty string on ring tones or dead air instead of hallucinating.

Training recipe

The encoder is thorhojhus's work. Both his released weights and his published recipe are used here:

• thorhojhus/whisper-large-v3-turbo-danish The encoder weights, and the Muon recipe below. Whisper-turbo fine-tuning.
• thorhojhus/multimodal-alignment The projector recipe.

His hyperparameters, used essentially unchanged:

The compensated update matters: plain bf16 write-back discards most of a 5e-6 Muon step, because the update is below the representable gap at typical parameter magnitudes. Kahan compensation preserves it (This is absolutely essential for MUON).

Changes from his setup: split learning rates (encoder Muon 5e-6, Qwen3 decoder 2e-5, projector AdamW 1e-4), the decoder replacement described above, and a projector-alignment phase that trains the projector alone against the frozen encoder and decoder before unfreezing all 1.236 B parameters. Trained on a single RTX 5090.

Inference details

Decoding uses a per-clip token cap of ceil(seconds × 8) + 12, a repeated-token loop stop, and n-gram collapse on the output.

Clips over 30 s are split at the midpoints of detected silences into pieces of at most 28 s, and the hypotheses are joined. Context does not cross a chunk boundary, so a name established early in a long recording will not inform a later chunk, and capitalisation restarts per chunk.

transcribe length-sorts chunks before batching, which matters because a batch decodes until its longest member finishes and pads mel to its longest member.

Limitations

• Danish only. Other languages were not trained or evaluated.
• No word-level timestamps, only per-chunk boundaries.
• No speaker diarization.
• No streaming; decoding is offline.
• Conversational speech remains the hardest case at 18.38 WER, and overlapping speech, crosstalk and far-field microphones degrade it further.
• Context is not shared across chunks in long-form audio.
• FTSpeech carries 57 % of the sampling weight, so formal monologue Danish is the strongest domain.
• Numbers and rare proper nouns are the most common residual error class.

Users deploying in high-stakes settings should measure accuracy on their own population rather than assuming it is uniform across accents, ages and recording conditions.

Files

model.safetensors        1.24 B parameters, bf16
config.json              nested whisper + qwen configs, frame stack, encoder position scheme
variant.json             checkpoint variant and step
saga2/model.py           encoder, projector, decoder, generation
saga2/encoder_pos.py     encoder position handling and the variable-length window forward
saga2/gpu_mel.py         log-mel on GPU, matching WhisperFeatureExtractor
saga2/decode.py          chunking, guards, batching
run.py                   command-line transcription

Citation

@misc{saga2-2026,
  title  = {saga-2: Danish speech recognition with a Whisper encoder and a Qwen3 decoder},
  author = {{Capacit A/S}},
  year   = {2026},
  url    = {https://huggingface.co/capacit-ai/saga-2-m}
}

Acknowledgements

thorhojhus, whose whisper-large-v3-turbo-danish the encoder in this model comes from, and whose Muon recipe and projector recipe the training follows. Alexandra Institute for FTSpeech and CoRal, Norwegian Language Bank for NST, and Rye-AI for the Danish ASR leaderboard.

License

CC BY-NC 4.0 from 2026-09-24. Free for research, personal and other non-commercial use with attribution to Capacit A/S. Commercial use requires a license from Capacit A/S: info@capacit.com.

Revisions published before 2026-09-24 (up to commit f1c014f383d59be1302c4eac350f6335a30b8f68) were released under Apache 2.0 and remain available under those terms.

Third-party components keep their own licenses: the Whisper encoder is MIT (OpenAI) and the Qwen3 decoder is Apache 2.0. See NOTICE and LICENSES/. Training corpora carry their own licenses.

Attribution

When you share this model or anything built on it, credit it as:

capacit-ai/saga-2-m by Capacit A/S (https://huggingface.co/capacit-ai/saga-2-m), licensed under CC BY-NC 4.0.

If you modified the model, say so. In papers, cite the BibTeX entry under Citation.