int_wakeup-zipformer_small-ctx20-ft4

Decides, turn by turn, whether a voice assistant should respond — without requiring a wake word. It reads the current utterance together with up to 20 preceding turns, so it can accept a follow-up like "and the one after that?" and reject a question two people are asking each other.

Same recipe as int_wakeup-whisper_base-ctx20-ft4, with a Zipformer encoder in place of Whisper. Trained and evaluated on TCLResearchEurope/intelligent_wakeup v1.0.1.

Results

Test split: 243 conversations, 3,647 scored turns, 6.72 h of recording of which 5.28 h is speech not addressed to the assistant. Threshold 0.35, swept on validation and applied unchanged to test.

False accepts per hour are over the 5.28 h of not-addressed speech; over the whole 6.72 h recording this model gives 3.12/hour.

By turn type:

It reaches the same F1 as the Whisper model by a different trade: better at spotting the wake word (0.981 vs 0.943), worse at follow-ups without one (0.716 vs 0.794). If your traffic is mostly explicit invocations, this is the better of the two; if it is mostly conversational follow-up, take the Whisper one.

Averaging this run's three kept checkpoints gives F1 0.895 at 5.31 FA/hour — better F1, worse false accepts.

Architecture

25.3M parameters; the encoder is 22.0M of them.

1. Per-turn encoder — Zipformer-small (6 stacks, 12 layers, 256-d output) over Kaldi filterbank features, then learned attention pooling to one 256-d vector per turn.
2. Context transformer — 4 causal layers, 256-d, over the last 20 turn vectors plus a speaker-role embedding each. Turn t attends to 0..t and never to the future.
3. Heads — a trigger logit, plus an auxiliary turn-type head used only as a training signal.

Initialised from icefall's LibriSpeech CR-CTC zipformer-small, then trained in two phases: encoder frozen with turn embeddings cached, then the top 4 encoder layers unfrozen (10.9M trainable).

Unlike Whisper, Zipformer runs on the utterance's real length rather than a padded 30 s window. Utterances are still capped at 30 s to match what Whisper saw, and floored at 0.3 s — below about 0.09 s the convolutional frontend returns zero frames.

Two encoders at inference — please read

Phase 2 fine-tuned the encoder for the current turn while the context turns kept being read from a cache built with the frozen encoder. The model expects that asymmetry, and inference has to reproduce it:

Encoding everything with the fine-tuned encoder raises no error and costs about 0.18 F1 (measured on the Whisper sibling). inference.py does it correctly; start from that rather than reimplementing. The context encoder's weights are downloaded from the Hub on first run.

This asymmetry is a training artefact rather than a design choice, and it is the first thing to fix in a retrain.

Usage

pip install -r requirements.txt
python inference.py session.wav turns.json

turns.json is the turn segmentation — this model classifies turns, it does not find them, so bring your own VAD and diarisation:

[{"start": 0.0,  "end": 4.2,  "speaker": "Tariq"},
 {"start": 4.2,  "end": 9.8,  "speaker": "Sigma"},
 {"start": 9.8,  "end": 13.1, "speaker": "Priya"}]

Audio is 16 kHz mono. Use the literal name "Sigma" for the assistant's own turns — they form context but are never scored. Speaker ids are roles, not identities: the assistant is 1, humans are 2, 3, 4… by order of first appearance within the conversation. inference.py handles that mapping.

It is a causal model, so the same code shape works for streaming: encode each turn as it ends, keep the last 20 context vectors, classify.

The encoder code under wakeup_model/zipformer/ is vendored from k2-fsa/icefall (Apache-2.0) and torchaudio (BSD-2), unmodified. k2 and torchaudio are not dependencies: the only things needed from them are the Swoosh activations and the Kaldi filterbank, both of which are pure PyTorch and included.

Training

Needs the training repo, which carries the data pipeline:

python precompute_embeddings.py 'splits=[train,validation,test]' \
  dataset.embedding_cache_dir=.cache/embeddings-zf
python train.py +experiment=int_wakeup-zipformer_small-ctx20-ft4 \
  dataset.embedding_cache_dir=.cache/embeddings-zf

training_config.yaml is that experiment file; resolved_config.yaml is the exact config this checkpoint was produced with.

The embedding cache must be one your own run built. It stores the output of a randomly-initialised pooling layer, so a cache from a different run is a different vector space — the training repo pins the pooling head to the cache directory and refuses to mix them.

Limitations

• contextual recall is 0.716, the weakest of the encoders tried. Follow-ups with no wake word are the remaining recall problem.
• 3.98 false accepts/hour is not low enough for always-on use — roughly 30 spurious activations a day in a living room.
• Turn segmentation is assumed. This model sits downstream of VAD and diarisation, and their errors land directly on its input. It also cannot decide until a turn has ended, so latency is bounded by endpointing.
• The corpus is synthetic — TTS voices over mixed room tone, one generation pipeline, English only. Splits are grouped so scenario variants never straddle them, but test is still unseen conversations, same distribution. Expect materially worse numbers on real recordings; treat these as a ceiling.
• The wake word is the fixed string "Sigma", baked into the training data.

Citation

If you use this code or dataset, please cite:

@inproceedings{sowanski2026intelligentwakeup,
  title={Training Intelligent Voice Assistant Wakeup with Controllable Synthetic Conversations},
  author={Sowa{\'n}ski, Marcin and Leszczy{\'n}ski, Kacper and Krzywicki, Kacper and Wodnicki, Krzysztof},
  year={2026},
}