LeBenchmark: A Reproducible Framework for Assessing Self-Supervised\n Representation Learning from Speech

Abstract

Self-Supervised Learning (SSL) using huge unlabeled data has been\nsuccessfully explored for image and natural language processing. Recent works\nalso investigated SSL from speech. They were notably successful to improve\nperformance on downstream tasks such as automatic speech recognition (ASR).\nWhile these works suggest it is possible to reduce dependence on labeled data\nfor building efficient speech systems, their evaluation was mostly made on ASR\nand using multiple and heterogeneous experimental settings (most of them for\nEnglish). This questions the objective comparison of SSL approaches and the\nevaluation of their impact on building speech systems. In this paper, we\npropose LeBenchmark: a reproducible framework for assessing SSL from speech. It\nnot only includes ASR (high and low resource) tasks but also spoken language\nunderstanding, speech translation and emotion recognition. We also focus on\nspeech technologies in a language different than English: French. SSL models of\ndifferent sizes are trained from carefully sourced and documented datasets.\nExperiments show that SSL is beneficial for most but not all tasks which\nconfirms the need for exhaustive and reliable benchmarks to evaluate its real\nimpact. LeBenchmark is shared with the scientific community for reproducible\nresearch in SSL from speech.\n