VITS2 Text-to-Speech on Natasha Dataset
Model Details
Model Description
This model is an implementation of VITS2, a single-stage text-to-speech system, trained on the Natasha dataset for the Russian language. VITS2 improves upon the previous VITS model by addressing issues such as unnaturalness, computational efficiency, and dependence on phoneme conversion. The model leverages adversarial learning and architecture design for enhanced quality and efficiency.
- Developed by: Jungil Kong, Jihoon Park, Beomjeong Kim, Jeongmin Kim, Dohee Kong, Sangjin Kim
- Shared by: LangSwap.app
- Model type: Text-to-Speech
- Language(s) (NLP): Russian
- License: MIT
- Finetuned from model: No
Model Sources
- Repository: VITS2 PyTorch Implementation
- Paper: VITS2 paper
Usage
This model was dedicated to be used with this repository. https://github.com/shigabeev/vits2-inference
Sample usage:
git clone git@github.com:shigabeev/vits2-inference.git
cd vits2-inference
pip install -r requirements.txt
python infer_onnx.py --model natasha.onnx --text "Привет! Я Наташа!"
Direct Use
The model can be used to convert text into speech directly. Given a text input in Russian, it will produce a corresponding audio output.
Downstream Use
Potential downstream applications include voice assistants, audiobook generation, voiceovers for animations or videos, and any other application where text-to-speech conversion in Russian is required.
Out-of-Scope Use
The model is specifically trained for the Russian language and might not produce satisfactory results for other languages.
Bias, Risks, and Limitations
The performance and bias of the model can be influenced by the Natasha dataset it was trained on. If the dataset lacks diversity in terms of dialects, accents, or styles, the generated speech might also reflect these limitations.
Recommendations
Users should evaluate the model's performance in their specific application context and be aware of potential biases or limitations.
How to Get Started with the Model
To use the model, users can follow the guidelines and scripts provided in the VITS2 PyTorch Implementation repository.
Training Details
Training Data
The model was trained on the Natasha dataset, which is a collection of Russian speech recordings.
Training Procedure
Preprocessing
Text and audio preprocessing steps, as mentioned in the repository README, were followed.
Training Hyperparameters
- Training regime: This can be filled with details such as learning rate, batch size, optimizer used, etc.
Summary
The VITS2 model demonstrates improved performance over previous TTS models, offering more natural and efficient speech synthesis.
Environmental Impact
You can fill in the details regarding the environmental impact, based on the compute resources used for training.
Technical Specifications
Model Architecture and Objective
The VITS2 architecture comprises of various improvements over the original VITS, including but not limited to speaker-conditioned text encoder, mel spectrogram posterior encoder, and transformer blocks in the normalizing flow.
Compute Infrastructure
Hardware
Single Nvidia RTX 4090
Software
- Python >= 3.11
- PyTorch version 2.0.0
APA:
Kong, J., Park, J., Kim, B., Kim, J., Kong, D., & Kim, S. (Year). VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design. [Journal/Conference Name], [pages].