nanites-smoky-speech-otis / nanites-smoky-speech-opal
Anonymized, gender-pooled Smoky Mountain English text-to-speech voices — the 1930s regional dialect of the Great Smoky Mountains (NC + TN) as recorded by Joseph Sargent Hall in 1939. Built on Fun-CosyVoice3-0.5B-2512 (Apache-2.0, from FunAudioLLM), fine-tuned separately per gender. Second generation of the Smoky voices: new randomly-sampled speaker pools on the CosyVoice 3 engine (first generation: Earl + Ethel on CosyVoice2).
Personas
The two pooled voices ship under Appalachian codenames so the identity is a persona, never a single Hall speaker:
- Otis (male) — 15 speakers, 265 clips, 40.0 min. Best-8 epochs
[10, 8, 9, 12, 13, 15, 11, 6]by val loss. Samples:sample/otis_0.wav,sample/otis_1.wav,sample/otis_2.wav(chatbot-written lines, same three lines as Opal for comparison). - Opal (female) — 15 speakers, 160 clips, 24.8 min. Best-8 epochs
[8, 14, 15, 13, 9, 12, 11, 16]by val loss. Samples:sample/opal_0.wav,sample/opal_1.wav,sample/opal_2.wav.
In code/config the voices keep functional names
smoky-speech-male_v2 / smoky-speech-female_v2 with speaker IDs
<|male_pool_v2|> / <|female_pool_v2|>; "Otis" and "Opal" are the
display/persona names. Hugging Face repos are persona-named:
Nanite-Labs/nanites-smoky-speech-otisNanite-Labs/nanites-smoky-speech-opal
Pools (random subsets, seed 20260925)
Male v2 is fully disjoint from Earl (0 overlap). Female v2 reuses 10 of
Ethel's speakers (only 6 unused female codes existed in the archive) plus
5 fresh ones (GladysHoyle, sr011am, sr038am, sr049bm, sr084am):
- male:
sr012bm, sr042am, sr042bm, sr048am, sr050bm, sr057bm, sr058am, sr062bm, sr064bm, sr067bm, sr074am, sr076am, sr076bm, sr102bm, sr161bm - female:
GladysHoyle, sr011am, sr011bm, sr033bm, sr038am, sr049am, sr049bm, sr060am, sr066bm, sr068am, sr075am, sr075bm, sr084am, sr086am, sr086bm
Pipeline per clip (same as v1): 16 kHz → 24 kHz resample, spectral-gate denoise, energy-VAD trim, SNR filter (≥ 8 dB).
Training
- Base:
FunAudioLLM/Fun-CosyVoice3-0.5B-2512(llm.pt + flow.pt + hift.pt) - Data: pooled chunk sets → kaldi-style dirs
(
wav.scp/text/utt2spk+ per-utteranceinstruct:You are a helpful assistant.<|endofprompt|>, required by the v3 LLM) → campplus embeddings + v3 speech tokens → parquet →cosyvoice/bin/train.py(--model llm, 1×GPU, bf16, constant-lr 1e-5,use_spk_embedding: True). Single speaker ID per gender. - 8 GB recipe: bitsandbytes AdamW8bit; whole model bf16 (no GradScaler);
gradient checkpointing (
COSYVOICE_GC=1); DDP wrap skipped at world_size 1; single-proc join guard. Patches live inexperts/smoky/train_sft.py(repo clone stays pristine). - Outputs:
llm_avg.ptper gender (best-8 by val loss); voice dirs assembled from the v3 base + averaged LLM + bakedspk2info.pt. - Inference note (v3): synthesis text must carry the instruct prefix —
You are a helpful assistant.<|endofprompt|>is prepended automatically byexperts/smoky/cosyvoice_infer.pyfor v3 voice dirs. SFT mode only (inference_sft, pooled embedding, no reference clip).
python experts/smoky/cosyvoice_infer.py \
--model_dir <voice-dir> \
--text "What was sharecropping like?" \
--sft_spk_id "<|male_pool_v2|>" \
--out out.wav
Evaluation
- WER (whisper
medium.en, 3 novel chatbot-written dialect lines, 132 words): Otis 6.1%, Opal 7.6%. Target < 12%. (Residual errors are dialect spellings: crick→creek, dawnin→dawning.) - Anonymization: pooled campplus embedding max cosine to the closest pool-member mean: 0.924 (male) / 0.927 (female) — a blended mean identity matching no single speaker (v1: 0.888/0.909).
- Sample lines were written by the collection's own chat model
(
nanites-smoky-1b-chat) so text and speech share the regional register.
Provenance
Audio: public-domain 1939 Hall field recordings (USC archive). Transcripts (Montgomery & Reed 2017, copyrighted) used locally only — never redistributed; training artifacts are derived phonetic tokens, not source text. Pooling is structural anonymization: one speaker ID per gender, no reference audio at inference.