Cycle-Consistent Speech Enhancement

Abstract

Feature mapping using deep neural networks is an effective approach for\nsingle-channel speech enhancement. Noisy features are transformed to the\nenhanced ones through a mapping network and the mean square errors between the\nenhanced and clean features are minimized. In this paper, we propose a\ncycle-consistent speech enhancement (CSE) in which an additional inverse\nmapping network is introduced to reconstruct the noisy features from the\nenhanced ones. A cycle-consistent constraint is enforced to minimize the\nreconstruction loss. Similarly, a backward cycle of mappings is performed in\nthe opposite direction with the same networks and losses. With\ncycle-consistency, the speech structure is well preserved in the enhanced\nfeatures while noise is effectively reduced such that the feature-mapping\nnetwork generalizes better to unseen data. In cases where only unparalleled\nnoisy and clean data is available for training, two discriminator networks are\nused to distinguish the enhanced and noised features from the clean and noisy\nones. The discrimination losses are jointly optimized with reconstruction\nlosses through adversarial multi-task learning. Evaluated on the CHiME-3\ndataset, the proposed CSE achieves 19.60% and 6.69% relative word error rate\nimprovements respectively when using or without using parallel clean and noisy\nspeech data.\n