VoiceCLAP-commercial Vocal-Burst Blend Scorer
A tiny, self-contained model that predicts, from a short speech clip containing a vocal burst (laugh, sob, gasp, sigh, groan, scoff, etc.), a 0-10 score for how naturally that vocal burst blends into the surrounding speech.
It is designed as a fast, cheap proxy reward for evaluating expressive TTS / voice-acting systems: does the non-verbal burst sound like an organic part of the performance, or does it sound spliced-in, robotic, or emotionally mismatched?
The package is fully standalone. It bundles the frozen VoiceCLAP-commercial audio embedder (weights + custom modeling code) and the trained blend head, so no other repositories are fetched at inference time — everything loads from local paths.
The bundled encoder is the frozen VoiceCLAP-commercial backbone; its weights are included in this repository for offline inference.
What it predicts — the 0-10 rubric
| Score | Meaning |
|---|---|
| 0 | The burst is disconnected: spliced/pasted-in, robotic, or the wrong emotion for the context (also assigned to clips with no genuine burst at all). |
| 5 | The burst fits the context but sounds performed / acted — mediocre delivery, audibly synthetic or stagey. |
| 10 | Totally organic: the burst is indistinguishable from a natural human reaction embedded in the speech. |
Intermediate values interpolate between these anchors. The output is
clamped to [0, 10].
Why the commercial embedding
This head sits on the 768-d VoiceCLAP-commercial audio embedding. That embedder is small (same 768-d width as VoiceCLAP-small) but much stronger for this task: the commercial-based head clearly beats the small-based one and comes within a hair of the ~4.7× wider large-v2 embedding, at a fraction of the cost.
| Embedder | Head | Blend width | Val MAE ↓ | Val corr ↑ |
|---|---|---|---|---|
| VoiceCLAP-commercial (this repo) | Linear(768,50)→GELU→Dropout(0.2)→Linear(50,1) | 768 | 2.057 | 0.625 |
| VoiceCLAP-small | Linear(768,50)→GELU→Dropout(0.2)→Linear(50,1) | 768 | 2.360 | 0.418 |
| VoiceCLAP-large-v2 | (large head) | 3584 | 1.895 | 0.645 |
Takeaway: on the same 768-d budget, moving from the small to the commercial embedding drops MAE from 2.360 → 2.057 and lifts correlation from 0.418 → 0.625 — essentially matching large-v2 (1.895 / 0.645) at 1/4.7 the embedding width.
Bonus: one encode, two scores
The same VoiceCLAP-commercial audio embedding also feeds the
genuineness predictor
laion/voiceclap-commercial-genuineness.
If you encode a clip once with the bundled commercial embedder you can run
both heads on that single 768-d vector — blend naturalness (this repo)
and burst genuineness — without a second forward pass through the encoder.
Architecture
16 kHz mono waveform
│
▼
VoiceCLAP-commercial audio encoder (frozen, 768-d embedding)
│ encode_waveform → 768-d
▼
L2-normalize (unit-norm, as in training)
│
▼
standardize: z = (emb − μ) / σ (μ, σ are 1×768 stats from training)
│
▼
Blend head (MLP):
Linear(768 → 50) → GELU → Dropout(0.2) → Linear(50 → 1)
│
▼
blend score ∈ [0, 10]
- Embedder: VoiceCLAP-commercial, a dual-tower CLAP-style model. Only
the audio tower (
encode_waveform) is used and it is kept frozen. It is bundled in this repo undervoiceclap_commercial/and loaded locally viatrust_remote_code=True— nothing is downloaded at inference time. - Head: a small MLP (≈38k params) trained on top of the frozen,
L2-normalized 768-d embeddings. The
μ/σstandardization stats are stored inside the checkpoint.
Usage
from blend_model import CommercialBlendScorer
# Loads the bundled VoiceCLAP-commercial embedder + blend head, all local.
scorer = CommercialBlendScorer(pkg_dir=".", device="cpu") # or device="cuda"
# Score a single wav (any sample rate / channel count; resampled to 16k mono):
score = scorer.score("clip_with_laugh.wav")
print(f"blend naturalness: {score:.2f} / 10")
# Score many at once:
scores = scorer.score_batch(["a.wav", "b.wav", "c.wav"])
# Score an in-memory waveform:
import torchaudio
wav, sr = torchaudio.load("clip.wav")
score = scorer.score_waveform(wav, sr)
Command-line:
pip install -r requirements.txt
python example.py clip_with_laugh.wav
# -> blend score (0-10): 7.41
Files in this repo
| File | Purpose |
|---|---|
blend_model.py | CommercialBlendScorer + BlendMLP inference code. |
blend_head_commercial.pt | Commercial blend head weights + μ/σ stats. |
voiceclap_commercial/ | Bundled frozen VoiceCLAP-commercial embedder (weights + modeling code). |
example.py | Minimal CLI example. |
requirements.txt | Python dependencies. |
Limitations
- Frozen encoder caps accuracy. Quality is upper-bounded by what the frozen VoiceCLAP-commercial audio embedding captures; the head cannot recover information the embedding discards.
- Mid-range scores are hardest. The model is most reliable at the extremes (clearly organic vs. clearly spliced/absent). Scores around the middle of the scale (4-6) carry the most uncertainty.
- Use as a fast proxy, not a judge. This is a lightweight reward / filtering signal (e.g. for ranking or reward-shaping TTS outputs). It is not a replacement for a strong multimodal judge or human evaluation on high-stakes decisions.
- Domain. Trained on speech clips with vocal bursts; behaviour on music, noise-only audio, or non-speech is undefined.
License
CC-BY-4.0. You are free to share and adapt this model, including for commercial use, provided you give appropriate credit to LAION. See https://creativecommons.org/licenses/by/4.0/.
Created and released by LAION.