Humaneness Ears: Base and Medium audio understanding

By Christoph Schuhmann · LAION · 7 October 2026 · CC BY 4.0

This repository releases the best completed two-epoch fine-tunes of LAION's encoder-only Whisper Base and Whisper Small multitask models. The release names are Humaneness Ears Base (Whisper Base backbone) and Humaneness Ears Medium (Whisper Small backbone). Given a recording, one forward pass predicts emotions, speaking style, sound quality, speaker-characteristic vectors and the location and type of non-speech vocal events such as laughter or crying. Both models are included: base/model.safetensors and medium/model.safetensors. The older training logs, evaluation JSON and the machine-readable listening manifest retain the small key as the historical Whisper backbone label; it refers to Humaneness Ears Medium.

These are audio encoders with task heads. There is no Whisper text decoder, ASR transcript, generated caption or language model. Gemini transcripts and captions helped prepare annotations; captions were not model inputs or an autoregressive training objective. CPS is a scalar prediction rather than a transcript. The requested separate MOSS caption LoRAs are outside this release and were not trained as part of these two runs.

Explore 1,000 real-audio examples in the Humaneness Ears atlas: 50 timed vocal bursts, 20-query Orange timbre and identity neighbor pages, 40 EmoNet and 57 VoiceNet top-ten rankings, and five-tier quality pages. Download the Gemini annotation dataset.

Open the full benchmark: Humaneness Ears Base/Medium versus CLAPv2 XS/M — human-label results, all 192 targets, Orange speaker-vector cosine and vocal-burst timing. Jump to the CLAP comparison.

Results and related releases

• Comprehensive English benchmark report: 36 frozen embedding probes, original and Humaneness Ears Base/Medium after Gemini tuning, and the two full CLAPv2 fine-tunes; human-label emotion benchmarks, all 192 score targets, speaker vectors, validation, burst timing and limitations.
• Which model is best for each task? includes genuineness and vocal-burst blend.
• Every target and validation result.
• Original layered Whisper weights: the Base S4 and original Whisper Small S3 checkpoints are the actual starting states for this release.

Medium has better Orange speaker-vector cosine and lower AudioBox/DNSMOS target error than Base, and a higher EmoNet intensity correlation. Base has slightly better CPS error, Flash frame F1 and VoiceNet-Emo mean rank correlation. A frozen probe can outperform both on other tasks: see the comprehensive report for all winners. This release does not claim one universal best model or statistical significance from small numerical differences.

What the models output

There are 192 jointly predicted scalar outputs plus the separate CPS head. A complete, ordered list of exact output keys and all their held-out results appears at the end of this card. The models return raw units, training-standardized values and display z-scores. Regression outputs are continuous and are not automatically clipped to the annotation's ordinal range.

Quick start: local CPU or GPU inference

Use Python 3.10 or newer. The measured production environment used Python 3.13.5, PyTorch 2.9.1, Transformers 5.14.1, SoundFile 0.14 and Safetensors. Install the appropriate PyTorch CPU/CUDA wheel for your machine; see requirements.txt for the other dependencies. Inference uses a custom model.py, not pipeline('automatic-speech-recognition') or WhisperForConditionalGeneration.from_pretrained.

Download inference assets without the large optimizer checkpoints:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="laion/humaneness-ears-base-medium",
    local_dir="humaneness-ears",
    allow_patterns=["*.py", "requirements.txt", "*.json", "base/*.json",
                    "medium/*.json", "base/model.safetensors", "medium/model.safetensors"],
)

cd humaneness-ears
pip install -r requirements.txt
python inference.py clip.wav --model medium --device cpu --threads 4 > medium_prediction.json
python inference.py clip.wav --model base --device cpu --threads 1 > base_prediction.json
python inference.py clip.wav --model medium --device cuda:0 --threads 4 \
  --include-frame-probabilities > gpu_prediction.json

Input recordings must be 0.1–30 seconds. SoundFile decodes the supplied audio, stereo is averaged to mono, and SciPy resamples to 16 kHz. The bundled Whisper frontend creates 80-bin log-mel features. Longer recordings need an explicit external chunking policy. No original OpenAI weights, teacher checkpoints or Gemini API access are required at inference.

Reuse the model when processing multiple files:

from inference import Predictor

predictor = Predictor(".", model_size="medium", device="cpu", threads=4)
result = predictor.predict("clip.wav", include_frames=True)
print(result["scores_raw"]["genuineness_0_6"])
print(result["scores_z"]["emo_Amusement"])
print(result["burst_event_proposals"])  # start_s, end_s, top3_classes

--frame-threshold and --event-threshold default to 0.5; --max-events defaults to 32. This matches the new report's proposal operating point. Lowering the event threshold can increase recall and false positives. Frame regions merge overlapping events, whereas the independent proposal head can represent them separately. Neither class softmax values nor sigmoid values have been calibrated as real-world confidence estimates.

Architecture

Base uses six encoder layers, hidden width 512, and 26,441,735 total parameters. Medium uses 12 encoder layers, hidden width 768, and 97,102,031 total parameters. These counts include our heads and exclude the absent Whisper decoder. The encoder alone contains 20,590,592 / 88,154,112 parameters respectively.

From the final encoder sequence we concatenate masked mean, minimum, maximum and standard deviation. Separately, each transformer layer contributes its masked mean vector. Shared per-family projection MLPs reduce these means to 32 or 64 dimensions, and a sample-specific softmax gate combines layer information. These features add residuals to the 192-score head. Speaker heads use 128D layer-mixture routes; CPS uses 32D. The frame head predicts occupancy; the proposal head predicts independent onset logits and log durations. A 64D layer-mixture route for event classification averages only frames inside that event from every layer, then combines them with the final-layer local mean and a 53-way classifier.

model.py is the exact architecture used in training. Exported weights preserve every saved tensor without quantization or conversion. The inherited Whisper configuration still contains some unused decoder fields; architectures and is_encoder_decoder are set for this encoder-only release. No decoder tensors are present. The custom inference code must be used.

Training data, annotation priority and splits

The selected pool contains 66,199 exact-audio unique clips / 197.36 hours after deduplication across the four selections. Valid Flash annotations exist for 65,163 clips / 194.17 hours. The 1,036 invalid or provider-blocked responses are excluded from this tuning stage. They were not converted into zero-emotion or no-burst targets.

Sources are the balanced S1 100-hour, 400-bucket selection from TTS Scaling Ladder DE/EN, LAION's Got Talent raw, balanced audio snippets 40×3k and voice annotation POC. Final selections preserve all source/task memberships while annotating each exact compressed-audio SHA256 once. Source membership counts are nonexclusive. Deterministic 90/5/5 hashing groups explicit speaker IDs, otherwise known synthetic audio families, otherwise exact audio hashes; observed counts differ from exact percentages because groups are kept intact.

training/provenance/valid_split_summary.json records valid counts, hours, language strings, memberships and the hash of the exact prepared target snapshot. The older TRAINING_READY.json also contains split counts including invalid responses; those are not the training counts above. Source revisions and original pool details are in training/provenance/selected_pool.json. The data, Gemini responses and teacher TARs are not redistributed by this model repository.

The recorded annotation model identifier is gemini-3.8-flash. Schema-valid Flash outputs take priority for 40 emotions, 57 VoiceNet dimensions, genuineness, blend, event counts, event start/end times and transcript-derived CPS. Corresponding VoiceCLAP emotion attributes also take the Flash value. Distinct constructs such as DNSMOS BAK or the original authenticity axis keep their own teachers. A null blend on a no-burst clip is masked, not replaced by an old blend score. Missing values remain masked. Speech-only objectives use domain masks; Orange speaker losses require valid known single-speaker clips. Teacher vectors are preserved even when their training loss is masked.

The prompt permits 78 vocal fine labels. Explicit aliases map compatible labels into the existing 53-class checkpoint vocabulary; unrepresented labels map to Other / rare burst. See gemini_burst_mapping.json for every mapping. Original fine labels and event captions remain in the annotation sidecars. End times may be clamped to the decoded duration within the accepted 0.3-second annotation tolerance, with original timing and the clamp flag retained in the data.

Exact fine-tuning procedure

Starting models were the lowest-validation-loss Base S4 and Whisper Small S3 snapshots from the original S1→S10 layered curriculum campaign. Each snapshot contains only the progress up to its own stage, rather than the final S10 weights: Base's original update was 266,238; Medium's was 237,434. The original campaign ran one epoch per nested stage with 10% P3 synthetic mixture exposures per stage and a continuous cosine schedule. Its full code and recorded run configs are included under training/ and training/original_curriculum/.

Gemini tuning strictly restored all encoder and head weights, started a fresh AdamW optimizer and cosine schedule, and unfroze the entire encoder, including its positional embedding table. This is full fine-tuning, not LoRA or a frozen-encoder probe.

There is no P3 replay in this extra two-epoch stage. A DDP sampler may pad a few examples to distribute batches equally; validation is evaluated completely on rank zero without duplicating its samples. The full config, class counts and epoch loss logs are under training/.

Losses

The implementation is training/train_layered_curriculum.py::loss_terms, reused unchanged by training/gemini_finetune/train_whisper.py:

• Masked Huber regression, delta 1, on standardized scalar targets; per-family weights are the exact GROUPS constants in training/multitask_p3_smoke.py.
• CPS: standardized Huber delta 1, weight 0.2.
• Speaker vectors: 0.2 × (cosine distance + 0.1 × Huber delta 0.1), only valid single speakers.
• Frames: weighted BCE (positive weight 5), weight 0.5, plus soft Dice weight 0.25.
• Onsets: weighted BCE (positive weight 100), weight 0.2.
• Event log duration: Huber delta 0.5, weight 0.1, evaluated at labeled starts.
• Event class: categorical cross entropy, weight 0.2, label smoothing 0.02; training-only class weights sqrt(median event count / class count), clipped to 0.25–4.

Every epoch saves latest and, when improved, best states. Published base/checkpoint.pt and medium/checkpoint.pt are the exact best files: model, AdamW, scheduler, four-rank RNG, completed epoch, update, normalization, taxonomy and original config. They support epoch-boundary continuation with the same DDP world size. Both runs already finished epoch 2; extending training requires an explicit new phase/schedule, not blindly resuming the completed two-epoch plan. Inference uses only Safetensors and never loads the .pt training pickle.

Normalization

raw_score = standardized_network_output × training_std + training_mean
display_z = (raw_score − original_training_teacher_median) / original_training_teacher_std
CPS_raw = CPS_network_output × CPS_training_std + CPS_training_mean

training_normalization.json is the original frozen checkpoint normalization, preserved through Gemini tuning. It is required for correct raw units. display_normalization.json is the train-only median/std reference from the original release, retained for compatible visualization and rankings. gemini_train_statistics.json separately stores train-only Flash mean/std/counts; its inherited “provisional” description refers to the incremental parser, but the file here is the completed final export. It was not substituted into network decoding. Display z-scores are relative context, not class probabilities or confidence intervals.

How the data were mounted and how to reproduce

Training ran on JUPITER Booster. /e/home, /e/project1, /e/scratch and /e/software were existing shared HPC filesystems visible to every rank; no Docker bind mount or network download occurred inside the GPU training. Source audio and same-audio teacher outputs stayed in WebDataset TARs. TarReader caches a bounded set of open archives and reads named members; it does not extract millions of individual files. Teachers are cached annotations, not called inside the gradient loop.

targets.jsonl contains audio_tar / audio_member and embedding tar / member references. On another machine, obtain the same complete prepared data, copy TARs and rewrite the absolute TAR prefixes to your local mounts while preserving member names and audio hashes. Provide checkpoint_normalization.json, gemini_burst_mapping.json and TRAINING_READY.json beside it. The repository includes the parser, schema prompt and teacher backfill code; recreating its data requires the upstream datasets, teacher weights and your own annotation service access. There is no credential embedded here. Source availability and terms still apply.

To reproduce the two-epoch experiment, download the original Base S4 or Whisper Small S3 .pt initializer from the previous repository, not this release's already-tuned state. Then:

pip install -r requirements-training.txt
python training/make_finetune_config.py --model base \
  --data-root /your/prepared --initial-checkpoint /your/original/base/S4/checkpoint.pt \
  --output-dir /your/new-base-run --config /your/base-run.json
python -m torch.distributed.run --standalone --nproc_per_node=4 \
  training/gemini_finetune/train_whisper.py --config /your/base-run.json

For the historical Whisper Small training run, use --model small and the original small/S3/checkpoint.pt. The config helper changes paths while preserving recorded hyperparameters. training/gemini_finetune/train_whisper.sbatch is the original submitted Slurm launcher, with original mount paths for reference. Evaluator scripts likewise preserve original study paths; adapt study_paths.py and benchmark data paths if reproducing the full comparison. Safe-weight export tools and verification receipts are included.

Held-out Flash target agreement

Both models are tested on the same 3,544 valid Flash test clips. These targets are machine annotations, not a fresh human panel. Speaker cosine uses 1,718 valid single-speaker clips; blend uses 2,391 clips with non-null blend. Family MAE is the unweighted mean over valid axes.

Normalized MAE is measured in frozen training standard deviations. Burst interval metrics use Hungarian matching of independent onset/duration proposals, not binary contiguous-frame regions. Boundary errors and predicted-span class accuracy are conditional on matches at IoU ≥0.1. Reference-span class accuracy gives the classifier the true interval; it is not end-to-end localization accuracy. Maximum proposals is 32, with a fixed 0.5 onset threshold. Event precision is low despite high recall; the report includes all IoU cutoffs and precision/recall.

The validation tables in evaluation/{base,small}/validation_metrics.json (historical backbone keys) use the existing 3,239 checkpoint-selection clips, not an untouched test. Supplementary evaluator loss uses unit class weights, while the training selection loss above used the configured class weights. Different loss values under those two policies are expected.

Public human-label benchmarks

EmoNet uses 12,000 mapped clips and fixed 0–4→0–10 endpoint scaling; two unsupported labels from the published 12,600-clip benchmark are omitted. VoiceNet-Emo uses 7,986 current-repository questions with at least two raters; VoiceNet-Ext uses 13,917. EXT and DIM refer to the same benchmark. Current snapshot coverage differs from paper coverage. The report includes unflagged, untruncated and alternate agreement cuts and explains optimistic oracle threshold numbers.

CREMA-D and RAVDESS figures are supervised matched nested actor-cross-validation score adapters, not zero-shot paper numbers. Each adapter sees the model's 192 raw predictions, a train-fold StandardScaler and a 64-unit GELU MLP: 12,742 / 12,872 parameters respectively. Five outer actor folds and three inner folds choose learning rate 0.001/0.003 and 20/50 epochs. Outer test actors never fit that fold's scaler or adapter. These benchmark-only adapters are not the model's burst classifier and are not automatically applied by inference.py. Full fold records, metrics and confusions are included; the report compares the same adapter budget across all backbones. Upstream training/source overlap with the public benchmarks was not exhaustively audited.

P3 retention and limitations

On the identical fixed 2,000-clip P3 test audit, frame F1 changes from 0.9622 to 0.8051 for Base and 0.9693 to 0.8392 for Medium after Gemini tuning. Flash test frame F1 rises from 0.3072/0.3171 to 0.6929/0.6892 respectively. This is a target/domain tradeoff: there was no P3 replay in this phase. All matched pre/post P3 validation/test JSONs are included. It is not valid to claim universal burst improvement from Flash agreement alone.

The selected pool is not a representative random sample of all natural audio. Quality scores imitate AudioBox/DNSMOS/Empathic teachers rather than measuring new human MOS. Speaker cosine measures regression agreement with Orange, not verification accuracy or human identity. The burst classifier groups unsupported fine labels, event proposals have many false positives, and emotional or expressive clips can differ from the synthetic P3 construction labels. Recordings longer than 30 seconds and multi-speaker embeddings require additional policies.

Annotation models, taxonomies and packages

Teacher inference runs on the exact selected waveform. New scores and float16 target vectors are persisted in SHA-bound companion TARs; the gradient loop reuses those sidecars. Their model revisions and target-level provenance remain in the prepared annotations.

Repository contents and export verification

No audio, source transcripts, raw Gemini responses or service credentials are published by this model repository. Model verification uses a real held-out waveform internally and saves only its hash and output-contract evidence. Evaluation JSONs contain aggregate benchmark results, prompts/classes and fold protocols; no listening audio is copied into this release.

License and attribution

The new fine-tuned weights, new code and documentation are released by LAION under Creative Commons Attribution 4.0 International. Credit Christoph Schuhmann and LAION, link this repository and indicate changes. This license does not relicense upstream source audio, teacher models or external packages. The original OpenAI Whisper components remain MIT; Orange's identity teacher retains its upstream CC BY-SA 3.0 terms. See NOTICE.md and upstream model/data cards for their notices.

@misc{schuhmann_humaneness_ears_2026,
  title = {Humaneness Ears: Base and Medium Encoder-Only Audio Understanding},
  author = {Schuhmann, Christoph and {LAION}},
  year = {2026},
  url = {https://huggingface.co/laion/humaneness-ears-base-medium}
}

All 192 scalar targets: exact held-out results

These are Flash test targets, with the remaining AudioBox/DNSMOS/Orange/Empathic families provided by their own teachers. Raw MAE uses each target's units. Norm. MAE uses its frozen training standard deviation. r is Pearson and ρ is Spearman. Missing metrics remain unavailable. The source JSON has separate counts for every model/target.

Canonical 53-class vocal-burst vocabulary

The classifier predicts these categories. Fine labels outside this vocabulary use the explicit aliases/fallbacks in gemini_burst_mapping.json; the fine-to-group23 map is also preserved.