mend-flan-t5-small
Turns spoken dictation into written text: removes filled pauses, resolves self-repairs, restores casing and punctuation.
send it to john no wait sarah um by friday
-> Send it to Sarah by Friday.
A fine-tune of google/flan-t5-small (77M) on 50,000 synthetically disfluent
sentences plus 25,000 real Switchboard utterances. Code, evaluation and the full
write-up: https://github.com/daltonoscar0/mend
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained("daltonoscar0/mend-flan-t5-small")
tok = AutoTokenizer.from_pretrained("daltonoscar0/mend-flan-t5-small")
text = "um send the the report to john no wait sarah by friday"
ids = tok("clean up: " + text, return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=160)[0], skip_special_tokens=True))
# Send the report to Sarah by Friday.
The clean up: prefix is required — it is the task prefix the model was trained
with. Input should be lowercased with edge punctuation removed, which is the
register the model saw in training.
Results
Scored on 293 held-out Switchboard conversations the model never trained on, and on a held-out split of the synthetic data. Brackets are 95% bootstrap intervals over 2000 resamples of sentences.
| System | Set | N | Exact match | WER (words only) | Filler removal F1 | Repair resolution |
|---|---|---|---|---|---|---|
| Copy baseline | real speech | 4000 | 0.325 | 0.117 | 0.000 | 0.000 |
| flan-t5-small, synthetic only | real speech | 4000 | 0.455 | 0.056 | 0.859 | 0.326 [0.285, 0.366] |
| this model | real speech | 4000 | 0.616 | 0.026 | 0.955 | 0.503 [0.460, 0.546] |
| Copy baseline | synthetic | 10000 | 0.030 | 0.132 | 0.000 | 0.000 |
| this model | synthetic | 10000 | 0.646 | 0.012 | 0.996 | 0.863 [0.848, 0.877] |
The headline finding of the project is the gap between the second and third rows. A model trained only on synthetically injected disfluencies resolves about a third of real human self-repairs; adding real speech to the training mix takes it to about a half. The two intervals do not overlap.
Limitations
Read these before using it for anything.
- Half of real self-repairs are still unresolved (0.503). This is not a solved problem.
- It has learned Switchboard's transcription conventions, including where that corpus puts commas and which discourse markers it treats as removable. Against a corpus that draws those lines elsewhere it will look worse.
- No real ASR output was involved. The disfluent side of every training pair is a transcript, never a recognition hypothesis, so the model has never seen a substitution error a real recogniser makes.
- Fragments are unhandled by construction. Utterances with cut-off words
("d-") were dropped from training, so
it sure does ma make a differencekeeps the strandedma. - Register mismatch. The target is dictation; the real training speech is telephone conversation.
Two real failures from the held-out set, for calibration:
in and then i it's like well the thrill is gone
gold And then it's like the thrill is gone.
out And then I, it's like, the thrill is gone. <- abandoned "i" kept
in degree it's of course it's a whole lot easier
gold Degree, of course, it's a whole lot easier.
out Degree, it's a whole lot easier. <- drops "of course"
Training
| Base | google/flan-t5-small (77M) |
| Data | 50,000 synthetic pairs + 25,000 real SwDA utterances (33% real) |
| Epochs | 2 |
| Batch | 32 |
| LR | 5e-5 |
| Hardware | Apple M-series, 62 minutes |
| Validation loss | 0.216 → 0.159 |
Synthetic pairs come from a disfluency injector structured after Shriberg's account of self-repair — reparandum, interregnum, repair — applied to WikiText sentences and dictation templates. Real pairs come from the Switchboard Dialog Act corpus, split by conversation so no conversation appears in both training and evaluation.
Licence
MIT. Switchboard Dialog Act corpus is freely redistributable; WikiText is CC BY-SA 3.0.