weights-and-wires/audio8-tts-from-scratch
A DualAR text-to-speech language model trained from scratch on
440 clips (72.6 minutes) of LibriSpeech
clean/train.100, 4 speaker(s).
⚠️ This is a training-pipeline experiment, not a production TTS model. The ~601 M-parameter transformer was randomly re-initialized and trained for 280 optimizer steps. Expect undertrained, partly unintelligible speech. LibriSpeech is 16 kHz upsampled to the codec's 44.1 kHz, which caps quality further.
What was trained vs. reused
| Component | Status |
|---|---|
| Slow AR transformer (24 layers, dim 896) | trained from random init |
| Fast AR transformer (4 layers) | trained from random init |
Neural codec (codec.pth, 10x4096 @ 44100 Hz) | reused frozen from Audio8/Audio8-TTS-Preview-0.6b |
| Tokenizer / architecture config | reused from the same checkpoint |
Training configuration
| Setting | Value |
|---|---|
| epochs | 10 |
| effective batch size | 16 |
| learning rate | 0.0002 (cosine, 0.05 warmup) |
| weight decay / grad clip | 0.01 / 1.0 |
| precision | fp32 weights + bf16 autocast |
| max sequence length | 2048 |
| train / eval examples | 440 / 23 |
| hardware | 1x NVIDIA A100-SXM4-40GB |
Final metrics
| Metric | Value |
|---|---|
| final train slow_loss | 3.9706 |
| final train fast_loss | 5.4779 |
| final train slow_accuracy | 0.4759 |
| final train fast_accuracy | 0.1876 |
| final eval slow_loss | 7.6749 |
| final eval fast_loss | 6.9021 |
Random-guess baselines for reference: slow_loss ln(vocab) = 11.96,
fast_loss ln(4096) = 8.32.
Usage
import soundfile as sf, torch
from transformers import AutoModel, AutoProcessor
repo = "weights-and-wires/audio8-tts-from-scratch"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).eval().cuda()
inputs = processor(text=["Hello world."], return_tensors="pt")
inputs = {k: v.cuda() for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=512, temperature=0.8, top_p=0.95,
top_k=50, do_sample=True, return_dict_in_generate=True)
audio, lengths = model.decode_audio(out.codes)
sf.write("out.wav", audio[0, :int(lengths[0])].float().cpu().numpy(), 44100)
The model was trained on the no-reference prompt template only — pass text= alone;
voice cloning via reference_audio= was never trained and will not work.
License
Apache-2.0, inherited from the base checkpoint. LibriSpeech is CC-BY-4.0.