Evaluating the Reliability of Acoustic Speech Embeddings

Robin Algayres, Mohamed Salah Zaïem, Benoît Sagot, Emmanuel Dupoux

2020 · 22 citations · 28 references

DOIFull text

Open access

Concepts

Abstract

Speech embeddings are fixed-size acoustic representations of variable-length\nspeech sequences. They are increasingly used for a variety of tasks ranging\nfrom information retrieval to unsupervised term discovery and speech\nsegmentation. However, there is currently no clear methodology to compare or\noptimise the quality of these embeddings in a task-neutral way. Here, we\nsystematically compare two popular metrics, ABX discrimination and Mean Average\nPrecision (MAP), on 5 languages across 17 embedding methods, ranging from\nsupervised to fully unsupervised, and using different loss functions\n(autoencoders, correspondence autoencoders, siamese). Then we use the ABX and\nMAP to predict performances on a new downstream task: the unsupervised\nestimation of the frequencies of speech segments in a given corpus. We find\nthat overall, ABX and MAP correlate with one another and with frequency\nestimation. However, substantial discrepancies appear in the fine-grained\ndistinctions across languages and/or embedding methods. This makes it\nunrealistic at present to propose a task-independent silver bullet method for\ncomputing the intrinsic quality of speech embeddings. There is a need for more\ndetailed analysis of the metrics currently used to evaluate such embeddings.\n

References

28