RelayS2S
Checkpoints for RelayS2S: Dual-Path Speculative Generation for Real-Time Dialogue.
A duplex speech-to-speech model that drafts a short response prefix speculatively and hands off to a cascaded ASR → LLM pipeline for the continuation. A small learned verifier decides whether the drafted prefix is safe to say out loud.
Code:
License note. Tagged
apache-2.0because the backbone is Qwen2.5-0.5B, which is Apache-2.0. Confirm this is the licence you intend for the trained weights and the speech encoder before relying on it.
Contents
| Path | What it is |
|---|---|
s2s/s2s.ckpt | Fast-path duplex S2S model (PyTorch Lightning checkpoint, 6.0 GB) |
tokenizer/ | Qwen2.5-0.5B tokenizer used by the S2S model |
verifier/best_model.pt | Prefix verifier (~170K params) |
verifier/tokenizer/ | Tokenizer the verifier uses to count prefix tokens |
pretrained/speech_encoder.pt | Conformer speech encoder, stage-0 initialisation |
pretrained/adapter.pt | CNN adapter, stage-0 initialisation |
s2s.ckpt is a training checkpoint: about 58% of it is Adam optimizer state that
inference does not use. The weights themselves are 2.79 GB.
Two things you need to know before using this
1. The model only detects end-of-turn over exact digital zero.
It was trained on synthesised dialogues whose inter-turn gaps are literally
0.0 — 100% zero samples. It learned that as the end-of-turn cue. Real
microphone audio never produces it: background noise at RMS 1e-5, about −100 dB
and completely inaudible, is enough to stop [BOS] firing entirely and
silently. There is no error; the model simply never responds.
Live microphone input therefore needs a noise gate that zeroes quiet frames before the model sees them. A per-frame RMS threshold of 0.01 with a two-frame hangover works. Dataset replay needs no gate.
2. The verifier threshold is 0.405, not 0.5.
That is the tuned value in verifier/inference_config.json, giving 65.6% recall
on bad prefixes at 96.0% acceptance of good ones — roughly a 9.6% discard rate.
Using 0.5 will discard more than intended.
Special tokens
The S2S tokenizer adds five control tokens to the Qwen vocabulary. They must be
registered before loading, or the embedding matrix will not match:
[SIL] (stay silent), [BOS] (start speaking), [EOS], [BOC] (backchannel),
[STP] (stop — user barge-in), [HOD].
The model consumes a 160 ms frame clock and emits exactly one token per frame.
Citation
@article{relays2s2025,
title = {RelayS2S: Dual-Path Speculative Generation for Real-Time Dialogue},
year = {2025},
eprint = {XXXX.XXXXX},
archivePrefix = {arXiv}
}