Anime-XCodec2-44.1kHz: A 44.1kHz Upsampling Variant of Anime-XCodec2

TL;DR: Anime-XCodec2-44.1kHz is a fine-tuned variant of NandemoGHS/Anime-XCodec2. It incorporates upsampling layers and RMS loss inspired by the Inworld TTS-1 paper to produce 44.1kHz output, trained on ~22k hours of Japanese speech.

Only the decoder was updated; the encoder and codebook remain frozen, so speech tokens are identical to the original XCodec2. This makes the model a drop‑in decoder for downstream systems that already work with XCodec2 tokens (e.g., Llasa).

🔗 Quick Links

• Demo (Gradio / Hugging Face Spaces): https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-44.1kHz-Demo
• This repository (44.1kHz fine-tune): NandemoGHS/Anime-XCodec2-44.1kHz
• Baseline 16kHz model: NandemoGHS/Anime-XCodec2
• Original XCodec2: HKUSTAudio/xcodec2
• Reference Paper (Inworld TTS-1): https://arxiv.org/abs/2507.21138
• Reference Implementation (Inworld TTS): https://github.com/inworld-ai/tts

1) Model Summary

• What it is: A neural speech codec based on Anime-XCodec2 (which is based on XCodec2), fine-tuned to output 44.1kHz high-fidelity Japanese speech (anime/game-style).
• Key Change: Integrates an UpSamplerBlock and utilizes RMS Loss (inspired by Inworld TTS-1) into the decoder architecture.
• Training scope: Decoder-only fine-tuning on ~22,000 hours of Japanese data. Encoder and codebook are frozen.
• Compatibility: Speech tokens are identical to HKUSTAudio/xcodec2 and NandemoGHS/Anime-XCodec2.
• Input Sampling rate: 16 kHz (for encoding, same as XCodec2).
• Output Sampling rate: 44.1 kHz (decoded audio).

2) Intended Use

• Decode XCodec2 speech tokens (e.g., from Llasa or other AR generators) into high-fidelity 44.1kHz Japanese speech (anime/game-style).
• Upgrade existing Anime-XCodec2 (16kHz) pipelines to 44.1kHz output.
• Audio Super-Resolution: As the model accepts 16kHz input and outputs 44.1kHz reconstructed audio, it can also be used as a form of audio super-resolution. However, its performance for this specific purpose is untested/unevaluated.

3) How to Use (Important)

This model modifies the original XCodec2 architecture by adding upsampler blocks. You MUST use the provided custom xcodec2 library fork for inference, as the standard library will not work.

• Installation:

  # Install the customi xcodec2 library
  pip install https://huggingface.co/NandemoGHS/Anime-XCodec2-44.1kHz/resolve/main/xcodec2-0.1.6.tar.gz
  

• Usage: Once the custom library is installed, you can load and use this model just as you would the original XCodec2 or Anime-XCodec2 models. The core inference logic remains the same.

For a complete, working code example, please refer to my Hugging Face Spaces Demo: https://huggingface.co/spaces/OmniAICreator/Anime-XCodec2-44.1kHz-Demo

4) Limitations & Trade-offs

• Language scope: Optimized for Japanese. Performance on other languages may degrade.
• Content domain: Tuned toward anime/game-style voices.
• Library Dependency: Requires the custom xcodec2 library linked above. It is not compatible with the original xcodec2 library.

5) Data (High-Level)

• ~22,000 hours of Japanese speech, with a focus on anime/game-style voices.
• Data was prepared for 44.1kHz target output during training.

6) Training Procedure (High-Level)

• Base Model: NandemoGHS/Anime-XCodec2 (16kHz)
• Architecture Modification:
  • Integrated the UpSamplerBlock from the Inworld TTS-1 implementation into the decoder.
  • Upsampler parameters: hop_length=147, upsample_factors=[3, 2], kernel_sizes=[7, 6].
• Loss Function: Adopted RMS Loss (Root Mean Square loss), as introduced in Inworld TTS-1, in addition to the original losses.
• Frozen: Encoder and Codebook (token compatibility preserved).
• Updated (fine-tuned): generator.backbone, generator.head, generator.upsampler, fc_post_a

7) Samples (A/B/C/D Listening)

Note: Original audio is 48 / 44.1 kHz. Baseline and Anime-XCodec2 are 16 kHz. This model outputs 44.1 kHz.

8) License

• CC-BY-NC 4.0 (inherited from XCodec2 and Anime-XCodec2).
• See: https://creativecommons.org/licenses/by-nc/4.0/

9) Acknowledgements

• HKUSTAudio/xcodec2 (Original model)
• Inworld AI for their work on Inworld TTS-1 (Upsampler architecture and RMS Loss).
• Thanks to contributors and the community around Japanese speech resources.