2020 · 22 citations · 28 references
ComputingComputer ScienceComputational LinguisticsInformation RetrievalNatural Language ProcessingAcoustic PhoneticsLanguage ModelingSpeech AcousticsSpeech RecognitionSpoken CorporaAutoencodersFrequency Distribution EstimationFrequency EstimationInformation Retrieval EvaluationMean Average PrecisionSegmentationSpeech Segmentation
Speech embeddings are fixed-size acoustic representations of variable-length\nspeech sequences. They are increasingly used for a variety of tasks ranging\nfrom information retrieval to unsupervised term discovery and speech\nsegmentation. However, there is currently no clear methodology to compare or\noptimise the quality of these embeddings in a task-neutral way. Here, we\nsystematically compare two popular metrics, ABX discrimination and Mean Average\nPrecision (MAP), on 5 languages across 17 embedding methods, ranging from\nsupervised to fully unsupervised, and using different loss functions\n(autoencoders, correspondence autoencoders, siamese). Then we use the ABX and\nMAP to predict performances on a new downstream task: the unsupervised\nestimation of the frequencies of speech segments in a given corpus. We find\nthat overall, ABX and MAP correlate with one another and with frequency\nestimation. However, substantial discrepancies appear in the fine-grained\ndistinctions across languages and/or embedding methods. This makes it\nunrealistic at present to propose a task-independent silver bullet method for\ncomputing the intrinsic quality of speech embeddings. There is a need for more\ndetailed analysis of the metrics currently used to evaluate such embeddings.\n
28
Scikit-learn: Machine Learning in Python
Fabián Pedregosa, Gaël Varoquaux, Alexandre Gramfort et al. · arXiv (Cornell University) · 2012 · 63.7K citations · Full text