Speaker-Aware Speech Enhancement with Self-Attention

Abstract

Speech enhancement aims to improve the intelligibility and quality of speech that is affected by noise. In this paper, we propose a novel speaker-aware speech enhancement (SASE) method that extracts speaker information using long short-term memory (LSTM) layers, and then uses a convolutional recurrent neural network (CRN) to embed the extracted speaker information. It is shown in a series of comprehensive experiments that only a few seconds of reference audio suffice for the proposed SASE method to perform better than LSTM and CRN baseline systems. The addition of a self-attention mechanism can further boost relevant speech-quality metrics.