talyno-whisper-small-finance
A LoRA fine-tune of openai/whisper-small, specialized for earnings-call and investor-meeting
speech recognition in English and Chinese. This is the merged PyTorch/safetensors checkpoint used
as the conversion source for passpoo/talyno-whisperkit-coreml,
the WhisperKit CoreML build shipped in the Talyno iOS app.
- Base model:
openai/whisper-small(244M params) - Fine-tuning method: LoRA (PEFT), encoder frozen, adapters on
q_proj/v_proj - Languages: English (
en), Chinese — Simplified + Traditional (zh) - Format: PyTorch / safetensors (merged adapter, not a bare PEFT checkpoint)
- License: Apache 2.0
Training data & methodology
Two sequential training phases on the same LoRA adapter, merged into the base model afterward:
| Phase | Language | Dataset | Samples | LoRA config |
|---|---|---|---|---|
| 1 | English | kensho/spgispeech (earnings-call audio) | 2,000 train / 200 eval | r=8, alpha=16 |
| 2 | Chinese | google/fleurs cmn_hans_cn (real recordings, CC-BY) + TTS-synthesised Traditional-Chinese financial-terminology sentences | 2,000 train / 200 eval | r=16, alpha=32 |
Key details:
- The encoder is frozen throughout training — only the decoder's attention projections receive LoRA adapters, preserving Whisper's general acoustic representations while adapting decoding to financial vocabulary and IR/analyst-Q&A phrasing (EPS, EBITDA, guidance, etc.).
- Chinese training data is a mix of real speech (FLEURS) for acoustic diversity and TTS-synthesised
sentences to guarantee coverage of financial terms that are rare in general speech corpora.
Traditional-Chinese vocabulary coverage in particular comes from the TTS set, since FLEURS has no
Taiwan Mandarin (
cmn_hant_tw) locale. - Before merging (
merge_and_unload()),generation_config.forced_decoder_idsandconfig.forced_decoder_idsare explicitly cleared. Whisper'sset_prefix_tokens(language=...)calls during training otherwise leave a language token baked into the merged model's generation config, which forces every inference to one language regardless of input.
Downstream conversion
This checkpoint is the conversion source for the WhisperKit CoreML build hosted at
passpoo/talyno-whisperkit-coreml, using
whisperkittools. Note: conversion currently requires
pinning transformers==4.42.4 in the conversion environment — newer transformers versions changed
past_key_values handling in a way that's incompatible with the whisperkittools decoder tracer (see
the finetune guide for details).
Datasets used (licensing)
- SPGISpeech — earnings-call transcription corpus; see dataset card for license terms.
- Google FLEURS
cmn_hans_cn— CC-BY licensed. - TTS-synthesised financial sentences (proprietary sentence set, synthesized via Google Cloud Text-to-Speech) — not derived from any third-party corpus.
Limitations
- English and Chinese only. Earlier development generations of this project experimented with a broader 6-language scope (adding German, Spanish, Japanese, Korean); the model actually published here was retrained and narrowed to EN + ZH for training-data quality and maintainability reasons.
- Optimized for earnings-call / investor-meeting register — not benchmarked on casual conversation or other domains.
License
Apache 2.0, consistent with the base openai/whisper-small model.