2023 · 15 citations · 16 references
Language StudiesLinguisticsAcousticsMachine LearningPhoneticsAcoustic PhoneticsArticulatory PhoneticsDeep LearningSpeech AcousticsSpeech RecognitionSpeech Sound ProductionArticulationArticulatory-acoustic RelationsElectromagnetic ArticulographyLearned Feature RepresentationsLinear ProbingSelf-supervised Speech Representation LearningSpeech Learning ModelVocal Tract Modeling
Recent self-supervised learning (SSL) models have proven to learn rich representations of speech, which can readily be utilized by diverse downstream tasks. To understand such utilities, various analyses have been done for speech SSL models to reveal which and how information is encoded in the learned representations. Although the scope of previous analyses is extensive in acoustic, phonetic, and semantic perspectives, the physical grounding by speech production has not yet received full attention. To bridge this gap, we conduct a comprehensive analysis to link speech representations to articulatory trajectories measured by electromagnetic articulography (EMA). Our analysis is based on a linear probing approach where we measure articulatory score as an average correlation of linear mapping to EMA. We analyze a set of SSL models selected from the leaderboard of the SUPERB benchmark [1] and perform further layer-wise analyses on two most successful models, Wav2Vec 2.0 [2] and HuBERT [3]. Surprisingly, representations from the recent speech SSL models are highly correlated with EMA traces (best: r =0.81), and only 5 minutes are sufficient to train a linear model with high performance (r =0.77). Our findings suggest that SSL models learn to align closely with continuous articulations, and provide a novel insight into speech SSL.
16
Librispeech: An ASR corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey et al. · 2015 · 6K citations
Computing, Computer Science, English +17
SUPERB: Speech Processing Universal PERformance Benchmark
Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang et al. · 2021 · 637 citations
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Jacob Kahn, Maude Rivière, Wenlong Zheng et al. · 2020 · 458 citations · Full text