DIAMOND

A sequence-to-sequence model for speech restoration via an autoregressive RQ-Transformer over neural audio codec tokens

Diamond turns degraded audio into near-studio 44.1 kHz speech. A bidirectional Transformer encodes the degraded mel-spectrogram; an autoregressive decoder with cross-attention predicts the tokens of a frozen neural audio codec (Descript Audio Codec, 9-book RVQ), which synthesizes the restored waveform.

Restoration is not denoising. A codec cuts the band above 8 kHz, clipping cuts the peaks, a lost packet zeroes the frame — no mask can return what the observation no longer contains. The missing content has to be generated from the surviving formants, which is why Diamond is a generative seq2seq model rather than a filter.

Diamond is trained from scratch — no pretrained backbone — and reaches the level of the strongest open restorers while remaining the best from-scratch restorer in its class.

Highlights

• Two-transformer read-out. A time-transformer models the frame sequence; a compact depth-transformer walks the 9 RVQ codebooks inside each frame. This restores the RVQ causal chain and distributes the output projection — where earlier models read all nine books out of a single frame vector with parallel heads.
• From scratch, 166.6M trainable params. No pretrained speech backbone, 63 GPU-hours of training for the released checkpoint.
• 44.1 kHz output. The frozen DAC decoder synthesizes full-band audio; sibilants and "air" above 8 kHz are regenerated, not merely passed through.
• Content is measured, not assumed. A generative restorer can sound clean while smearing words, so CER against the ground-truth transcript is a mandatory second axis alongside DNSMOS.
• Calibrated degradation. Inputs come from a DSP augmentor whose codec palette is fitted per-sample to a real degraded-speech distribution, not from a bag of random effects.

Hear the difference

Real, heavily degraded speech restored to 44.1 kHz. DNSMOS-P.835 OVRL rises by more than a full point on each clip — audible even without headphones. Try it live in the Diamond Space.

Model Details

• Developed by: nineninesix.ai
• Model type: Autoregressive sequence-to-sequence speech restoration (encoder + RQ-Transformer decoder over codec tokens)
• Encoder: Bidirectional Transformer, no downsampling — 20 layers, 512d, 8 heads (63.9M)
• Decoder / time-transformer: causal self-attention + cross-attention — 20 layers, 512d, 8 heads (88.7M)
• Codebook read-out / depth-transformer: local AR over the 9 RVQ codes of a frame — 4 layers, 384d, 6 heads (14.0M)
• Parameters: ≈ 166.6M trainable (the DAC codec, ~74M, is frozen and downloaded at runtime)
• Audio codec: Descript Audio Codec — 44.1 kHz, 9-codebook RVQ, 86 frames/s
• Primitives: RMSNorm, learnable per-layer RoPE, QK-Norm, LayerScale, GELU FFN
• Input / output sample rate: any input, resampled to 24 kHz internally → 44,100 Hz output
• Training data: 681.5 h of studio speech, ~2.5k speakers, 28% natively wide-band.
• Languages: English
• License: Apache 2.0

Benchmarks

Measured on 750 real degraded recordings — identical clips for every model. DNSMOS-P.835 (raw sig_bak_ovr.onnx) for perceptual quality, CER against the ground-truth transcript for content preservation.

Diamond improves ≈88% of clips (mean ΔOVRL +0.255) and places second of six on perceptual quality — ahead of RE-USE, which trains on roughly four times the data. Both systems that beat it on CER are non-autoregressive: the gap is the exposure bias inherent to AR decoding, not a capacity limit. The one other autoregressive system here, UniSE, lands on exactly Diamond's mean CER (0.131) by a different route — which suggests the tail belongs to the paradigm, not to this particular recipe.

Links

• Checkpoint: huggingface.co/nineninesix/diamond-1.0
• Technical Report: TECH_REPORT.pdf
• Training: github.com/nineninesix-ai/diamond-train
• Website: nineninesix.ai

What it's good for

Offline restoration and dataset cleansing — turning large volumes of degraded recordings (podcasts, interviews, archival and user-generated audio that has passed through lossy codecs) into material clean enough to train on. This is the task Diamond was built for and the one it is measured on.

Keep in mind:

• A thin CER tail. As an autoregressive decoder, Diamond occasionally smears words on hard clips. Inference safeguards (chunked decoding, repetition penalty) hold it down, but the tail is real — check the transcript when content fidelity is critical.
• Decoding is serial, not real-time. One frame at a time, 86 frames per second of audio, plus the depth pass. This is offline restoration, not a live filter.
• English-centric. Training data is English; other languages are untested.
• It generates, it does not filter. Content above the codec's cutoff is invented from context — plausible, not recovered. Do not treat the output as forensic evidence of what was said.

Use it responsibly. Restored speech is synthesized speech. Don't present it as an unaltered recording, and don't use it to fabricate or misattribute what someone said.

Acknowledgments

Built on the Descript Audio Codec for the frozen token space. Trained on LibriTTS-R, Hi-Fi TTS, VCTK. Evaluated with DNSMOS P.835 and faster-whisper.

Citation

If you use this work in your research, please cite:

@software{diamond_2026,
  author = {Almaz Zholdoshbek uulu, Ulanbek Abdurazakov, Denis Pavlov and Nursultan Bakashov},
  title = {Diamond: A Sequence-to-Sequence Model for Speech Restoration via an Autoregressive RQ-Transformer over Neural Audio Codec Tokens},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nineninesix/diamond-1.0}},
  note = {Trained from scratch; no pretrained backbone}
}

References

@inproceedings{kumar2023dac,
  title={High-Fidelity Audio Compression with Improved RVQGAN},
  author={Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan},
  booktitle={NeurIPS},
  year={2023},
  note={arXiv:2306.06546}
}

@inproceedings{lee2022rqtransformer,
  title={Autoregressive Image Generation using Residual Quantization},
  author={Lee, Doyup and Kim, Chiheon and Kim, Saehoon and Cho, Minsu and Han, Wook-Shin},
  booktitle={CVPR},
  year={2022},
  note={arXiv:2203.01941}
}

@article{defossez2024moshi,
  title={Moshi: a speech-text foundation model for real-time dialogue},
  author={D{\'e}fossez, Alexandre and Mazar{\'e}, Laurent and Orsini, Manu and Royer, Am{\'e}lie and P{\'e}rez, Patrick and J{\'e}gou, Herv{\'e} and Grave, Edouard and Zeghidour, Neil},
  journal={arXiv preprint arXiv:2410.00037},
  year={2024}
}

@inproceedings{copet2023musicgen,
  title={Simple and Controllable Music Generation},
  author={Copet, Jade and Kreuk, Felix and Gat, Itai and Remez, Tal and Kant, David and Synnaeve, Gabriel and Adi, Yossi and D{\'e}fossez, Alexandre},
  booktitle={NeurIPS},
  year={2023},
  note={arXiv:2306.05284}
}

@inproceedings{koizumi2023librittsr,
  title={LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus},
  author={Koizumi, Yuma and Zen, Heiga and Karita, Shigeki and Ding, Yifan and Yatabe, Kohei and Morioka, Nobuyuki and Bacchiani, Michiel and Zhang, Yu and Han, Wei and Bapna, Ankur},
  booktitle={Interspeech},
  year={2023},
  note={arXiv:2305.18802}
}

@inproceedings{reddy2022dnsmos,
  title={DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors},
  author={Reddy, Chandan K. A. and Gopal, Vishak and Cutler, Ross},
  booktitle={ICASSP},
  year={2022},
  note={arXiv:2110.01763}
}

License

Apache 2.0 — this model and its weights are released under the Apache License 2.0.

The frozen Descript Audio Codec is downloaded at runtime and carries its own license (MIT).