Sotto 350M, ScribeKey tune, GGUF
A fine-tune of juanquivilla/sotto-cleanup-lfm25-350m
at revision 6df6f019170b8b55333c047b901886a51750a965, for on-device transcript cleanup with
llama.cpp. ScribeKey uses it for Smart Cleanup on Android.
It cleans up dictation the way Sotto does, and also applies spoken self-corrections that Sotto leaves in, and punctuates spoken lists:
- "send the draft to alex i mean jordan before lunch" becomes "Send the draft to Jordan before lunch."
- "um we need milk eggs bread and uh coffee" becomes "We need milk, eggs, bread and coffee."
Lists are written without a serial comma.
| File | Quantisation | Bytes | SHA-256 |
|---|---|---|---|
sotto-scribekey-350m-Q8_0.gguf | Q8_0 | 379,215,776 | cd1929420e0c013c070a2f2cd12f95bfb42c955697589b4a056d498cbc8235c9 |
Prompt
The same as Sotto's:
<|startoftext|>### Input:
{transcript}
### Output:
The model was trained with the tokenizer's BOS (<|startoftext|>). Either tokenize the string
above with special-token parsing and no automatic BOS, or drop <|startoftext|> from the text and
let llama.cpp add the BOS. Decode greedily and stop at EOS (<|im_end|>) or at ###; the cleaned
text is everything before ###, trimmed.
How it was made
- Data: 2,066 training and 236 validation rows generated from ScribeKey's own synthetic dictation templates: fillers, repeats, self-corrections, spoken numbers, money, times and dates, names and product names to keep, questions and commands that must be cleaned, not answered or carried out, and spoken lists of two to four items, some with a corrected item. No third-party corpus and no text written by another model.
- Training: a LoRA (rank 16, alpha 32) on every LFM2 projection, 2 epochs, batch 8, learning rate 2e-4 with cosine decay, fp32 on CPU, using the exact prompt above. The adapter is merged into the weights.
- Export:
convert_hf_to_gguf.pyfrom the llama.cpp tree vendored in llama-cpp-python 0.3.36 to BF16, thenllama_model_quantizeto Q8_0.
ScribeKey evaluation
CleanBench v1 is ScribeKey's hidden benchmark: 3,366 English dictations spoken by text-to-speech and transcribed by three speech models. Scores are for the text the app inserts: ScribeKey's safety guard replaces any output that brings in a word, value or negation the speaker did not say with rule-based cleanup.
| Sotto 350M Q8_0 | Previous revision (b13e7b2) | This revision, Q8_0 | |
|---|---|---|---|
| Right text inserted, all rows | 21.6% | 26.6% | 28.0% |
| Right text inserted, self-corrections | 2.1% | 10.1% | 13.9% |
| Right text inserted, questions to leave unanswered | 12.1% | 17.0% | 19.7% |
| Right text inserted, little to change | 58.0% | 61.1% | 62.7% |
| Inserted text with a value the reference lacks | 3.9% | 3.5% | 3.4% |
| Latency p50 / p95 (x86, 4 threads) | 249 / 619 ms | 228 / 568 ms | 212 / 515 ms |
On 30 hand-written spoken lists, this revision writes 29 exactly right; the previous revision wrote 12 and Sotto 5.
A second judge, TypeSafe's Jev, compared each output with the reference on hand-written dictations: this revision says the same thing more often than Sotto (63% against 58%) and adds or drops content no more often (6% against 6%).
Known gaps: a correction can occasionally be applied to the wrong word, and a transcript the speech model got wrong can still come back wrong.
Licence
The base model, LFM2.5-350M-Base, is licensed by Liquid AI, Inc. under the LFM Open License v1.0. Its grant for commercial use applies only to organisations with annual revenue under US$10 million. The Sotto fine-tune is published by juanquivilla under the MIT licence. This model is a derivative work of both and is distributed under the LFM Open License v1.0; see NOTICE for attribution and changes.