tidelganesh/svara-tts-v1-bf16
bf16 re-upload of kenpath/svara-tts-v1. Weights cast from the original fp32/fp16 checkpoint to bfloat16; no quantization-aware training, no calibration. Quality is lossless relative to the upstream model.
Why bf16
| fp32 (upstream) | bf16 (this repo) | |
|---|---|---|
| VRAM | ~32 GB | ~16 GB |
| Throughput vs fp32 | 1.0× | ~1.8–2.2× on Ampere/Ada/Hopper |
| Quality | baseline | indistinguishable |
| Bitsandbytes / calibration | n/a | not required |
bf16 is the sweet spot for serving TTS on a single 24 GB+ GPU. If you need to push VRAM further, see the int8 sister repo.
Hardware
| GPU | bf16 fit | Notes |
|---|---|---|
| A100 40/80 GB | ✅ comfortable | best throughput |
| A40 48 GB | ✅ very comfortable | plenty of headroom for concurrency |
| L4 24 GB | ✅ tight | use attn_implementation="flash_attention_2" |
| RTX 4090 24 GB | ✅ tight | same |
| T4 16 GB | ❌ | use int8 instead |
| RTX 3090 24 GB | ⚠️ | may OOM on long contexts; use int8 |
Software: CUDA ≥ 11.8, PyTorch ≥ 2.3, transformers >= 4.45, accelerate, flash-attn (strongly recommended).
Usage
import torch
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained(
"tidelganesh/svara-tts-v1-bf16",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="cuda",
attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained(
"tidelganesh/svara-tts-v1-bf16",
trust_remote_code=True,
)
trust_remote_code=True is required because the upstream model ships custom modeling code in modeling_svara.py.
Serving
This repo is intended to back a self-hosted Svara-TTS FastAPI server. Point the server's model loader at this repo instead of the upstream one:
- model = AutoModel.from_pretrained("kenpath/svara-tts-v1", trust_remote_code=True)
+ model = AutoModel.from_pretrained(
+ "tidelganesh/svara-tts-v1-bf16",
+ trust_remote_code=True,
+ torch_dtype=torch.bfloat16,
+ device_map="cuda",
+ attn_implementation="flash_attention_2",
+ )
Server contract (POST /v1/text-to-speech, 24 kHz PCM16 mono) is unchanged.
How it was made
pip install -U "transformers>=4.45" accelerate "huggingface_hub[cli]" hf_transfer
pip install -U flash-attn --no-build-isolation
export HF_HUB_ENABLE_HF_TRANSFER=1
huggingface-cli login
export HF_TOKEN=hf_xxx...
python backend/quantize_and_push_svara.py # ~10–20 min on A40
python backend/verify_svara_bf16.py # round-trip check
The script:
- Loads
kenpath/svara-tts-v1in bf16 with Flash-Attention 2. - Casts any stray fp32 parameters (custom code occasionally re-introduces them) back to bf16.
- Asserts every parameter is bfloat16.
- Saves with
safe_serialization=True(safetensors) sharded at 2 GB. - Copies the upstream
tokenizer/preprocessor/*.pyfiles sotrust_remote_codestill works. - Pushes everything via
model.push_to_hub+tokenizer.push_to_hubto this repo.
Limitations
- bf16 is not a quantization (int8/int4); it's a precision downcast with the same numeric range as fp32. There is no measurable quality loss.
- Custom modeling code is re-shipped from the upstream repo verbatim. If
kenpath/svara-tts-v1updates itsmodeling_svara.py, this repo must be re-pushed. - Not a fine-tune. The model is identical to upstream except for dtype.
Provenance
- Base model:
kenpath/svara-tts-v1 - Conversion: bf16 cast + safetensors re-serialization
- Tooling:
transformers,accelerate,bitsandbytes(not used for this repo),huggingface_hub,flash-attn
Related
tidelganesh/svara-tts-v1-bnb-8bit— bitsandbytes int8 weights (~50% of bf16 VRAM, minimal quality loss on TTS, listen-test before production).
License
Inherits the upstream license. Please refer to kenpath/svara-tts-v1 for terms.