The 2020 ESPnet Update: New Features, Broadened Applications, Performance Improvements, and Future Plans

TLDR

ESPnet is an end‑to‑end speech processing toolkit that has rapidly expanded to cover a wide range of speech applications. The project began in December 2017 to advance end‑to‑end speech recognition and now aims to provide up‑to‑date speech processing tools for researchers to develop technologies collaboratively. ESPnet now includes TTS, VC, ST, and SE with beamforming, separation, denoising, and dereverberation, training all applications end‑to‑end and enabling further integration and joint optimization. ESPnet offers reproducible all‑in‑one recipes that achieve state‑of‑the‑art performance on various benchmarks by employing transformers, advanced data augmentation, and conformer models.

Abstract

This paper describes the recent development of ESPnet (https://github.com/espnet/espnet), an end-to-end speech processing toolkit. This project was initiated in December 2017 to mainly deal with end-to-end speech recognition experiments based on sequence-to-sequence modeling. The project has grown rapidly and now covers a wide range of speech processing applications. Now ESPnet also includes text to speech (TTS), voice conversation (VC), speech translation (ST), and speech enhancement (SE) with support for beamforming, speech separation, denoising, and dereverberation. All applications are trained in an end-to-end manner, thanks to the generic sequence to sequence modeling properties, and they can be further integrated and jointly optimized. Also, ESPnet provides reproducible all-in-one recipes for these applications with state-of-the-art performance in various benchmarks by incorporating transformer, advanced data augmentation, and conformer. This project aims to provide up-to-date speech processing experience to the community so that researchers in academia and various industry scales can develop their technologies collaboratively.