2023 · 16 citations · 14 references
ComputingComputer SystemsAcousticsComputational AcousticsNatural Language ProcessingDeep LearningSound SynthesisSpeech RecognitionSpeech SynthesisText GenerationVoice GenerationEnd-to-end Speech RecognitionEnd-to-end Speech SynthesisInverse Fourier TransformNeural Speech SynthesisShort-time Fourier TransformSpeech Waveform GenerationText-to-speech Synthesis
We propose a lightweight end-to-end text-to-speech model using multi-band generation and inverse short-time Fourier transform. Our model is based on VITS, a high-quality end-to-end text-to-speech model, but adopts two changes for more efficient inference: 1) the most computationally expensive component is partially replaced with a simple inverse short-time Fourier transform, and 2) multi-band generation, with fixed or trainable synthesis filters, is used to generate waveforms. Unlike conventional lightweight models, which employ optimization or knowledge distillation separately to train two cascaded components, our method enjoys the full benefits of end-to-end optimization. Experimental results show that our model synthesized speech as natural as that synthesized by VITS, while achieving a real-time factor of 0.066 on an Intel Core i7 CPU, 4.1 times faster than VITS. Moreover, a smaller version of the model significantly outperformed a lightweight baseline model with respect to both naturalness and inference speed. Code and audio samples are available from https://github.com/MasayaKawamura/MB-iSTFT-VITS.
14
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
WaveNet: A Generative Model for Raw Audio
Aäron van den Oord, Sander Dieleman, Heiga Zen et al. · arXiv (Cornell University) · 2016 · 3.6K citations · Full text
Computing, Musicology, Acoustics +16
Statistical parametric speech synthesis using deep neural networks
Heiga Zen, Andrew Senior, Mike Schuster · 2013 · 831 citations