Automatic Multimedia Knowledge Discovery, Summarization and Evaluation

Abstract

This paper presents novel methods for automatically discovering, summarizing and evaluating multimedia knowledge from annotated images in the form of images clusters, word senses and relationships among them, among others. These are essential for applications to intelligently, efficiently and coherently deal with multimedia. The proposed methods include automatic techniques (1) for constructing perceptual knowledge by clustering the images based on visual and text feature descriptors, and discovering similarity and statistical relationships between the clusters; (2) for constructing semantic knowledge by disambiguating the senses of words in the annotations using WordNet and the images clusters, and finding semantic relations between the senses in WordNet; (3) for reducing the size of multimedia knowledge by clustering similar concepts together; and (4) for evaluating the quality of multimedia knowledge using information and graph theory notions. Experiments show the potential of integrating the analysis of images and annotations for improving the performance of the image clustering and the wordsense disambiguation, the importance of good concept distance measures for knowledge summarization, and the usefulness of automatic measures for evaluating knowledge quality.