Investigating RNN-based speech enhancement methods for noise-robust Text-to-Speech

TLDR

Noisy speech degrades TTS voice quality, and while pre‑training enhancement helps, clean‑speech‑trained voices remain superior. The study compares two RNN‑based speech enhancement methods for training TTS systems. Both methods train an RNN to map noisy acoustic features to clean ones; one uses full TTS features (MCEP, aperiodicity, F0), the other uses only MCEP from the magnitude spectrum, then reconstructs waveforms for TTS training. The MCEP‑only approach yields higher F0 accuracy and, subjectively, synthetic voices rated as high as those trained on clean speech.

Abstract

The quality of text-to-speech (TTS) voices built from noisy speech is compromised. Enhancing the speech data before training has been shown to improve quality but voices built with clean speech are still preferred. In this paper we investigate two different approaches for speech enhancement to train TTS systems. In both approaches we train a recursive neural network (RNN) to map acoustic features extracted from noisy speech to features describing clean speech. The enhanced data is then used to train the TTS acoustic model. In one approach we use the features conventionally employed to train TTS acoustic models, i.e Mel cepstral (MCEP) coefficients, aperiodicity values and fundamental frequency (F0). In the other approach, following conventional speech enhancement methods, we train an RNN using only the MCEP coefficients extracted from the magnitude spectrum. The enhanced MCEP features and the phase extracted from noisy speech are combined to reconstruct the waveform which is then used to extract acoustic features to train the TTS system. We show that the second approach results in larger MCEP distortion but smaller F0 errors. Subjective evaluation shows that synthetic voices trained with data enhanced with this method were rated higher and with similar to scores to voices trained with clean speech.