Whisper Base and Small: emotion, voice, quality and vocal-burst understanding

Latest two-epoch Gemini-tuned Whisper Base and Small

Download the current Gemini-tuned encoder-only models and complete code. By Christoph Schuhmann, released under LAION, CC BY 4.0. Both best checkpoints are from epoch 2 of full encoder/head tuning on 58,380 valid Flash training clips, with 3,239 validation and 3,544 test clips. The new repository contains safe inference weights, exact optimizer/scheduler/RNG states, train/display normalization, all training and inference code, source provenance and detailed held-out results. It starts from the original Base S4 and Small S3 stage checkpoints; the original weights remain available separately.

Comprehensive comparison and task winners includes genuineness, blend, emotions, voice style, speaker vectors, audio quality and vocal-burst timing. Flash-target agreement improves while P3 burst retention decreases; both domains are documented.

LAION · encoder-only models · 192 scalar outputs · two Orange speaker-vector heads · vocal-burst frames, count, timing and class · literal characters per second. These models fine-tune the complete encoder of OpenAI Whisper Base or Whisper Small. There is no Whisper decoder and no ASR transcription. Explore the 60-clip listening gallery or upload your own audio in the TTS-AGI CPU demo. The gallery examples are explanatory; the independent test below uses all eligible test clips.

The heads learn to approximate stored annotation-model outputs. A high predicted emotion score is not a clinical assessment, a human rating, or a calibrated probability. For rank comparisons, use the supplied training-reference z-scores alongside raw values. Event-class softmax values are likewise uncalibrated.

CREMA-D and RAVDESS supervised score adapters

Read the detailed English adaptation audit or the updated comprehensive report. Base S4 and Small S3 remain frozen; the adapters learn from their 80 scalar emotion outputs, with no hidden-state fitting.

These are post-hoc supervised benchmark adaptations, distinct from paper zero-shot results. Each scored actor is excluded from that adapter's training and inner selection. Separate downloadable all-data refits have no independent test score of their own. Upstream pretraining/teacher overlap is unverified. The report explains fitting counts, fairness limits and published specialized classifier comparisons. JSON adapters, cached predictions, fitting/inference code and reproducibility instructions. The gallery/upload demo continue to illustrate final S10 checkpoints; these corpus-specific adapters apply to the evaluated S4/S3 checkpoints.

What the models return

The complete 192-target order, valid test counts and per-dimension MAE/correlation are in the target table and training_normalization.json.

Which checkpoints are included

For each model, the three stages with the lowest fixed validation selection loss are preserved as both safe inference weights (model.safetensors) and exact resumable checkpoint.pt files. The latter include AdamW moments, scheduler, rank RNG state and data hashes; load pickle-based .pt files only from a trusted source. S10 is also included as the final-stage reference. The rank-one Base S4 and Small S3 checkpoints have now been evaluated on the source-disjoint P3 test and public benchmarks; the other two top-three stages per model have not. The validation selection metric is the sum of weighted P3 and Ladder validation losses; it is not a raw-score MAE or an accuracy percentage.

The full best-stage trajectory, including the six stages not packaged, is in evaluation/WHISPER_BASE_SMALL_LAYERED_S1_S10_RESULTS_2026-10-03.html and training/{base,small}_run_config.json. S10 is the default in the demo and inference script so that its outputs correspond to the results below.

Public benchmark audit for the rank-one checkpoints

The rendered English report and its repository HTML copy evaluate Base S4 and Small S3 on VoiceNet-Emo, VoiceNet-Ext, EmoNet-Voice Bench, CREMA-D, RAVDESS, the source-disjoint P3 test, Ladder validation, and a separate 300-clip vocal-burst locator validation mix. The report includes every target, speaker cosine, 53 burst classes, 20 ms frame metrics, timed-event IoU, benchmark label mappings, paper comparison rows, and machine-readable results.

The CREMA-D and RAVDESS rows use the revised v2 crosswalk across the 40 EmoNet heads. The earlier single-head acted-speech figures are superseded; the scoring source and complete crosswalk are included for audit.

The public benchmark questions differ from each other and from the model's direct regression outputs. In particular, the current emolia-dim repository has 13,917 non-LANG questions with at least two recorded raters, while VoiceNet Table 1 reports 16,166; its paper rows are therefore contextual rather than same-row comparisons. The EmoNet HF snapshot has 42 queried labels, of which 40 map to this model's emotion heads. The separate locator mix reveals substantially weaker burst transfer than the P3 test (Base/Small frame F1 0.263/0.279 versus 0.960/0.969 on P3 at threshold 0.5). See the HTML for exact protocols and limits.

Gemini S10 annotation smoke test

The 25-clip self-contained listening page compares Gemini 3.8 Flash audio-only annotations with existing S10 teacher labels and these models' S10 predictions. It includes original audio, transcripts, audio/director captions, all 40 EmoNet and 57 VoiceNet scores, and time-aligned vocal-burst annotations. The selected clips come from training tier 0; this is a qualitative target-audit example, not a held-out benchmark. Validated Gemini JSONL and agreement summaries are linked on the page.

Architecture

The pretrained Whisper encoder is trained end to end. For each encoder transformer layer, the model pools valid time frames and learns a gated layer mixture. Per-family residual experts with 32 or 64 hidden dimensions supplement a direct final-layer pooled route. The speaker heads receive 128-dimensional layer-mixture features; the separate CPS head receives 32-dimensional features. A 20 ms frame head predicts burst occupancy. An onset plus log-duration proposal head finds potentially overlapping events. For each proposal, the 53-way class head pools only frames inside that event from every encoder layer and combines them with final-layer event features. This is implemented in model.py and training/layered_multitask_model.py.

Inference uses mono 16 kHz audio lasting 0.1–30 seconds, an 80-bin Whisper log-mel frontend, and the corresponding base/config.json or small/config.json. The released model.safetensors files contain the complete fine-tuned encoder and heads. They do not need a second download of the original OpenAI weights.

Quick start

pip install -r requirements.txt
python inference.py clip.wav --repo-dir . --model small --stage S10 > prediction.json
python inference.py clip.wav --repo-dir . --model base --stage S10 > prediction_base.json

# A top-three validation checkpoint can be chosen explicitly:
python inference.py clip.wav --repo-dir . --model small --stage S3 > prediction_s3.json

inference.py returns all raw scores and display z-scores, CPS, both Orange vectors, burst count, binary regions and independent event proposals with top-three class softmax values. Add --include-frame-probabilities for the full 20 ms timeline. The Python entry point is predict(audio, repo_dir, model_size='small', stage='S10').

raw_score = network_score × training_std + training_mean
display_z = (raw_score − training_teacher_median) / training_teacher_std

The two statistics files are intentionally separate. training_normalization.json is required to recover teacher units from the trained network. display_normalization.json uses train-only reference medians and standard deviations to make scores with different baselines easier to compare. A display z-score is contextual, not a probability or confidence interval. The VoiceNet tempo score, VoiceCLAP talking speed, and literal CPS are separate targets.

Training data and procedure

Training used LAION's TTS Scaling Ladder DE/EN in the nested order S1→S10, one pass through each stage. A frozen 1,120,000-example P3 synthetic vocal-burst pool supplied 10% of the exposures within each stage; its 1,031,454-example training split was reused between stages. P3 validation contained 10,000 clips, the source-disjoint P3 test 9,693 clips, and 68,853 overlap candidates were excluded. The pool combines speech, clean burst sources, selected effects and music with exact inserted-event metadata and cached teacher sidecars. The source-disjoint test still consists of synthetic mixtures. No P3 test component was used as a training component; the release does not claim performance on all unrelated real-world speech.

Each model used four JUPITER Booster nodes with four GH200 GPUs per node (16 GPUs), global batch 256 (16 clips/GPU, accumulation 1) and 281,006 optimizer updates. AdamW used weight decay 0.01; peak encoder learning rate was 2×10⁻⁵ for Base and 1.5×10⁻⁵ for Small, while peak head learning rate was 3×10⁻⁴ for both. After 5,620 warmup updates, one cosine schedule continued across all ten stages without restarts and ended at 10% of each peak. Training used BF16 autocast and gradient clipping at 1.0. The exact run settings, sampler and dataset hashes are in training/{base,small}_run_config.json and the code under training/.

All 192 scalar scores and literal CPS use masked Huber regression on standardized targets. Missing or invalid annotation values are masked, never silently turned into zero labels. Burst frames use weighted binary cross entropy plus soft Dice; independent event starts use weighted BCE and log-duration Huber; the 53-class event head uses class-balanced cross entropy with label smoothing. Orange timbre and identity use cosine distance plus Huber against valid single-speaker vectors. See training/train_layered_curriculum.py for the exact weights and masks. Source paths in the scripts refer to the original JUPITER layout and must be adapted on another system.

Annotation models, taxonomies and attribution

These are the teachers and source taxonomies whose recorded outputs formed the targets. The released Whisper models are students; inference does not call these teachers.

Teacher scores are model predictions, not independent human ground truth. Speaker-vector outputs are trained to imitate the Orange target embeddings, not to establish a person's identity. Source licenses vary; see the linked upstream cards.

Final S10 reference evaluation on source-disjoint P3

Protocol: best S10 checkpoint for each model; 9,693 P3 source-disjoint test mixtures; 2,048 fixed Ladder validation clips for CPS, with 1,932 valid literal transcript-derived CPS targets. Score-family means are unweighted averages across valid dimensions. Normalized MAE divides the raw absolute error by the training standard deviation for that dimension; Pearson r measures linear agreement with the recorded teacher score. The 192-dimensional table below gives per-target counts because label masks differ (for example, 4,112 valid blend scores). Orange cosine uses 6,809 clips with valid single-speaker targets.

Detection, speaker and timing

Scalar family agreement with recorded teachers

Each row below is the unweighted mean across valid dimensions.

Frame F1 uses 20 ms bins and threshold 0.5. Binary-region F1 matches contiguous occupied spans at IoU ≥ 0.5. Onset/offset errors are calculated only on matched binary regions; overlapping inserted events can merge in the binary representation. The independent proposal/class metric instead matches separately predicted onset/duration proposals to the 12,071 inserted events at IoU ≥ 0.5 and requires the class to agree. The difference between true-span and proposal macro F1 is the main current limitation for typed burst localization. All details, including per-class results and checkpoint trajectory, are in evaluation/WHISPER_BASE_SMALL_LAYERED_S1_S10_RESULTS_2026-10-03.html and the four machine-readable JSONs under evaluation/.

Files

Limitations and license

The held-out P3 test is source-disjoint within the synthetic mixture pipeline; it does not quantify domain transfer to arbitrary recordings. Multi-speaker clips are masked for Orange teacher-vector evaluation. The model accepts at most 30 seconds at a time. Emotion and quality scores reproduce teacher annotations and should not be treated as factual statements about people. Class-probability outputs have not been calibrated, and 53-way proposal classification remains substantially weaker than binary burst localization.

LAION releases its fine-tuned weights, new code and documentation under CC BY 4.0. Attribute LAION and link this repository. The starting OpenAI Whisper components are MIT-licensed. Upstream teacher models and source audio retain their own terms; for example the Orange identity teacher is CC BY-SA 3.0. The Scaling Ladder card describes its mixed source licenses. The Space's audio samples are presented for inspection with provenance and are not relicensed through this model card.

All 192 scalar targets

The following rows are from the independent P3 test. Raw MAE is in each teacher's own scale; normalized MAE and Pearson make cross-dimension comparison easier. R_quality has no valid P3 test labels and is shown as unavailable, although the head was trained where that target was present.

Citation

@misc{laion_whisper_layered_multitask_2026,
  title = {Encoder-Only Whisper Base and Small for Emotion, Voice, Quality, Speaker Embeddings and Vocal-Burst Localization},
  author = {{LAION}},
  year = {2026},
  url = {https://huggingface.co/laion/whisper-base-small-layered-audio-scores}
}