int_wakeup-whisper_base-ctx20-ft4
Decides, turn by turn, whether a voice assistant should respond - without requiring a wake word. It reads the current utterance together with up to 20 preceding turns, so it can accept a follow-up like "and the one after that?" and reject a question that two people are asking each other.
Trained and evaluated on
TCLResearchEurope/intelligent_wakeup
v1.0.1.
Results
Test split: 243 conversations, 3,647 scored turns, 6.72 h of recording of which 5.28 h is speech not addressed to the assistant. Threshold 0.90, chosen on validation and applied unchanged.
| precision | recall | F1 | false accepts / hour | |
|---|---|---|---|---|
| this model | 0.931 | 0.853 | 0.890 | 3.6 |
| same model, encoder frozen (no Phase 2) | 0.824 | 0.816 | 0.820 | 9.9 |
GPT Realtime (gpt-realtime-mini) prompted baseline | 0.098 | 0.997 | 0.179 | 516.6 |
False accepts per hour are over the 5.28 h of not-addressed speech. Over the whole 6.72 h recording the figure is 2.8/hour.
By turn type:
| type | n | recall |
|---|---|---|
direct - contains the wake word | 158 | 0.937 |
contextual - follow-up, no wake word | 141 | 0.759 |
non-assistance - human-to-human | 3,348 | 19 false accepts |
direct and contextual are all positives, so only recall is meaningful for
them; non-assistance is all negatives.
Architecture
24.0M parameters.
- Per-turn encoder - Whisper-base encoder (6 layers, 512-d) over the turn's audio, then learned attention pooling to one 512-d vector.
- Context transformer - 4 causal layers, 256-d, over the last 20 turn vectors plus a speaker-role embedding per turn. Turn t attends to 0..t and never to the future.
- Heads - a trigger logit, and an auxiliary turn-type head used only as a training signal.
Trained in two phases: Phase 1 with the encoder frozen and turn embeddings
precomputed (3.4M trainable), then Phase 2 with the top 4 encoder layers
unfrozen (16.0M trainable). Phase 2 is what the -ft4 suffix means, and it is
where the accuracy comes from - it lifted direct recall from 0.867 to 0.937 by
teaching the encoder to actually hear the wake word.
Two encoders at inference - please read
Phase 2 fine-tuned the encoder for the current turn while the context turns kept being read from a cache built with the frozen encoder. The model therefore expects those two to be different, and inference has to reproduce it:
| encoder | |
|---|---|
| current turn | the fine-tuned one, in model.safetensors |
| context turns | pristine openai/whisper-base + context_pooling_head.safetensors |
Encoding everything with the fine-tuned encoder - the natural thing to write -
raises no error and drops validation F1 from 0.896 to 0.718, with
contextual recall collapsing from 0.759 to 0.290. inference.py does it
correctly; start from that rather than reimplementing.
This asymmetry is a training artefact rather than a design choice, and it is the first thing to fix in a retrain.
Usage
pip install -r requirements.txt
python inference.py session.wav turns.json
turns.json is the turn segmentation - this model classifies turns, it does not
find them, so bring your own VAD or diarisation:
[{"start": 0.0, "end": 4.2, "speaker": "Tariq"},
{"start": 4.2, "end": 9.8, "speaker": "Sigma"},
{"start": 9.8, "end": 13.1, "speaker": "Priya"}]
Audio is 16 kHz mono. Use the literal name "Sigma" for the assistant's own
turns - they form context but are never scored. Speaker ids are roles, not
identities: the assistant is 1, humans are 2, 3, 4… by order of first
appearance within the conversation. inference.py handles that mapping.
It is a causal model, so the same code shape works for streaming: encode each turn as it ends, keep the last 20 context vectors, classify.
Training
Needs the training repo, which carries the data pipeline. The recipe is one config:
python precompute_embeddings.py 'splits=[train,validation,test]' \
dataset.embedding_cache_dir=.cache/embeddings-v2
python train.py +experiment=int_wakeup-whisper_base-ctx20-ft4 \
dataset.embedding_cache_dir=.cache/embeddings-v2
training_config.yaml here is that experiment file; resolved_config.yaml is
the exact config this checkpoint was produced with.
The settings that mattered, measured:
| validation F1 | |
|---|---|
| Phase 1 baseline | 0.819 |
pos_weight 4.0 → 2.0 | 0.831 |
| + Phase 2, top 4 layers unfrozen | 0.899 |
| whisper-small instead of base | 0.818 - no gain for 4x the encoder |
| mean+max pooling instead of attention | 0.870 - worse |
Phase 2 delivers most of its gain in the first epoch or two after the switch. One caveat: this run was still improving when it hit its 50-epoch limit (0.881 → 0.889 → 0.891 → 0.899 over its last four epochs), so it is not a converged number.
The embedding cache must be one your own run built. It stores the output of a randomly-initialised pooling layer, so a cache from a different run is a different vector space - the training repo pins the pooling head to the cache directory and refuses to mix them.
Limitations
contextualrecall is 0.759. Follow-ups with no wake word are the whole remaining recall problem, and nothing tried so far moved them.- 3.6 false accepts/hour is not low enough for always-on use. A living room would see ~30 spurious activations a day.
- The corpus is synthetic - TTS voices over mixed room tone, one generation pipeline, English only. Splits are grouped so scenario variants never straddle them, but test is still unseen conversations, same distribution. Expect materially worse numbers on real recordings; treat these as a ceiling.
- Turn segmentation is assumed. Errors in your VAD land directly on this model's input.
- The wake word is the fixed string "Sigma", baked into the training data.
Citation
If you use this code or dataset, please cite:
@inproceedings{sowanski2026intelligentwakeup,
title={Training Intelligent Voice Assistant Wakeup with Controllable Synthetic Conversations},
author={Sowa{\'n}ski, Marcin and Leszczy{\'n}ski, Kacper and Krzywicki, Kacper and Wodnicki, Krzysztof},
year={2026},
}