pVAD v1 — speaker-conditioned frame VAD for telephony barge-in
A 171k-parameter streaming model that labels every 10 ms of an 8 kHz caller audio stream as silence, the enrolled caller, or someone else (a bystander, a TV, the agent's own echo). It is conditioned on a 192-d speaker embedding of the caller taken from the first seconds of the call, so a voice-agent's barge-in logic can ignore other voices in the room without waiting for a transcript.
Trained by Telenow AI on its own Indian PSTN call recordings (Hindi, English and other Indian languages) with synthetic bystander / TV / echo mixtures. No audio is released with this model.
Results (held-out recordings, synthetic mixtures, 1.5 s windows)
| window | cosine speaker gate (baseline) | pVAD v1 |
|---|---|---|
| caller alone → accepted as caller | 22 % | 96 % |
| caller + bystander overlapping → accepted | 11 % | 81 % |
| bystander only → accepted (false accept) | 0 % (+19 % unsure) | 6 % |
| window EER | 0.22 | 0.091 |
Frame accuracy 0.87, speech-frame AUC 0.97, target recall 0.84 at 5 % false accept. One 10 ms frame costs ~0.08 ms single-threaded in onnxruntime.
Files
| file | what |
|---|---|
pvad_stream_v1.onnx | streaming graph — one 10 ms frame per call, carries its own state (use this in production) |
pvad_full_v1.onnx | whole-sequence graph feats [1,T,40], emb [1,192], h0 [2,1,96] → logits [1,T,3] (offline scoring) |
pvad_v1.pt | PyTorch state_dict of the model below |
pvad_norm.json | feature mean / sd (already baked into both ONNX graphs; here for retraining) |
config.json | the full I/O and feature contract |
example_stream.py | reference streaming inference with onnxruntime |
train_log.txt | the training run's validation curve |
Model
fbank(40) → Linear(40→64) + ReLU
→ causal Conv1d(64, k=5) + ReLU
→ causal Conv1d(64, k=5, dil=2) + ReLU
→ FiLM(emb 192 → scale, shift) (x · (1+γ) + β)
→ GRU(64→96, 2 layers)
→ Linear(96→3) [silence, caller, other]
Normalisation (feats − mu) / sd with the stored global statistics is the
first op inside both graphs.
Input features
Kaldi-style log-mel filterbank, 8 kHz, 40 bins, 25 ms Hamming window,
10 ms hop, no dither, DC removal, 0.97 pre-emphasis, 256-point FFT, energies
floored at float epsilon then ln, no mean subtraction, on int16-scale
samples (i.e. waveform × 32768). Equivalent to:
torchaudio.compliance.kaldi.fbank(x_int16_scale[None], num_mel_bins=40,
frame_length=25, frame_shift=10, dither=0.0, energy_floor=0.0,
sample_frequency=8000, window_type="hamming", use_energy=False, snip_edges=True)
Conditioning (enrollment) embedding
A 192-d, L2-normalised WeSpeaker ECAPA-TDNN512-LM embedding
(voxceleb_ECAPA512_LM.onnx, CC-BY-4.0) of 1.5–3 s of the caller's clean
speech: audio upsampled 8 → 16 kHz, 80-bin Kaldi fbank (25/10 ms, Hamming, no
dither), per-utterance mean subtraction, feats [1,T,80] → embs [1,192]. The
model was trained with enrollments of 1.5–3 s (first turn) and 3 s (later in
the call), 50/50. Any other speaker encoder requires retraining.
Streaming contract (pvad_stream_v1.onnx)
| tensor | shape | meaning |
|---|---|---|
feats | [1,1,40] | the new 10 ms frame |
hctx | [1,4,64] | last 4 post-ReLU input-hidden frames (zero-init) |
cctx | [1,8,64] | last 8 conv-1 outputs (zero-init) |
emb | [1,192] | enrollment embedding |
h0 | [2,1,96] | GRU state (zero-init) |
→ logits | [1,1,3] | silence / caller / other |
→ hn | [2,1,96] | next GRU state |
→ h1 | [1,1,64] | push into hctx (drop the oldest) |
→ c1 | [1,1,64] | push into cctx (drop the oldest) |
The two rings reproduce the training graph's causal zero padding exactly: streaming output equals the whole-sequence graph to ~2e-6 from the first frame.
Intended use and limits
- Built for barge-in gating in a full-duplex voice agent on telephony audio: "is this speech the caller's?" before STT has returned. It says who is speaking, not whom they address.
- Evaluated on synthetic bystanders (other real callers, room-filtered and mixed at 3–22 dB below the caller, TV at 12–25 dB below, agent echo); real-room recordings were not part of the evaluation.
- Trained on 8 kHz Indian-language telephony; wideband or non-telephony audio is out of distribution.
- Needs an enrollment; until the caller has spoken ~1.5 s there is nothing to condition on — pass through until then.
- Two people legitimately sharing a call must be handled by policy on top (e.g. admit a non-caller voice after N speech runs inside a window).