LRSpeech

TLDR

Speech synthesis and recognition require large text‑speech corpora, but most of the world’s 6,000+ languages lack sufficient data, hindering low‑resource TTS and ASR development. This work introduces LRSpeech, a low‑resource TTS and ASR system designed to support rare languages with minimal data. LRSpeech combines pre‑training on high‑resource languages, fine‑tuning on low‑resource data, a dual TTS–ASR transformation that iteratively improves both, and knowledge distillation to tailor TTS to a target speaker and enhance ASR across voices. Experiments on English and Lithuanian show LRSpeech delivers >98 % intelligibility and MOS > 3.5 for TTS, promising ASR accuracy, and operates with extremely low‑resource training data, and it is already being deployed in a commercial cloud speech service.

Abstract

Speech synthesis (text to speech, TTS) and recognition (automatic speech recognition, ASR) are important speech tasks, and require a large amount of text and speech pairs for model training. However, there are more than 6,000 languages in the world and most languages are lack of speech training data, which poses significant challenges when building TTS and ASR systems for extremely low-resource languages. In this paper, we develop LRSpeech, a TTS and ASR system under the extremely low-resource setting, which can support rare languages with low data cost. LRSpeech consists of three key techniques: 1) pre-training on rich-resource languages and fine-tuning on low-resource languages; 2) dual transformation between TTS and ASR to iteratively boost the accuracy of each other; 3) knowledge distillation to customize the TTS model on a high-quality target-speaker voice and improve the ASR model on multiple voices. We conduct experiments on an experimental language (English) and a truly low-resource language (Lithuanian) to verify the effectiveness of LRSpeech. Experimental results show that LRSpeech 1) achieves high quality for TTS in terms of both intelligibility (more than $98%$ intelligibility rate) and naturalness (above 3.5 mean opinion score (MOS)) of the synthesized speech, which satisfy the requirements for industrial deployment, 2) achieves promising recognition accuracy for ASR, and 3) last but not least, uses extremely low-resource training data. We also conduct comprehensive analyses on LRSpeech with different amounts of data resources, and provide valuable insights and guidances for industrial deployment. We are currently deploying LRSpeech into a commercialized cloud speech service to support TTS on more rare languages.