Audio Set: An ontology and human-labeled dataset for audio events

TLDR

Audio event recognition, the human‑like ability to identify and relate sounds from audio, remains a nascent problem in machine perception, unlike image detection which has benefited enormously from comprehensive datasets such as ImageNet. This paper introduces Audio Set, a large‑scale dataset of manually annotated audio events designed to bridge the data gap between image and audio research. Audio Set is built using a carefully structured hierarchical ontology of 632 audio classes, with human labelers annotating 10‑second YouTube video segments selected through metadata, context, and content‑based searches. The resulting dataset, unprecedented in breadth and size, is expected to substantially stimulate the development of high‑performance audio event recognizers.

Abstract

Audio event recognition, the human-like ability to identify and relate sounds from audio, is a nascent problem in machine perception. Comparable problems such as object detection in images have reaped enormous benefits from comprehensive datasets - principally ImageNet. This paper describes the creation of Audio Set, a large-scale dataset of manually-annotated audio events that endeavors to bridge the gap in data availability between image and audio research. Using a carefully structured hierarchical ontology of 632 audio classes guided by the literature and manual curation, we collect data from human labelers to probe the presence of specific audio classes in 10 second segments of YouTube videos. Segments are proposed for labeling using searches based on metadata, context (e.g., links), and content analysis. The result is a dataset of unprecedented breadth and size that will, we hope, substantially stimulate the development of high-performance audio event recognizers.