A generic framework of user attention model and its application in video summarization

TLDR

Video redundancy makes automatic extraction of essential content crucial for managing large libraries, and human attention provides an effective mechanism for prioritizing information, motivating a user attention model for video indexing based on importance ranking. This paper presents a generic user attention framework that estimates viewer attention to video content and demonstrates its application in video summarization without full semantic understanding or complex heuristic rules, showcasing its effectiveness, robustness, and generality. The framework defines viewer attention through visual, aural, and partial semantic perceptions and proposes modeling methods for visual and aural attentions to estimate attentions viewers may pay to video contents. User studies on video summarization show that the attention model provides an alternative way to video understanding, achieving promising results that demonstrate its effectiveness, robustness, and generality.

Abstract

Due to the information redundancy of video, automatically extracting essential video content is one of key techniques for accessing and managing large video library. In this paper, we present a generic framework of a user attention model, which estimates the attentions viewers may pay to video contents. As human attention is an effective and efficient mechanism for information prioritizing and filtering, user attention model provides an effective approach to video indexing based on importance ranking. In particular, we define viewer attention through multiple sensory perceptions, i.e. visual and aural stimulus as well as partly semantic understanding. Also, a set of modeling methods for visual and aural attentions are proposed. As one of important applications of user attention model, a feasible solution of video summarization, without fully semantic understanding of video content as well as complex heuristic rules, is implemented to demonstrate the effectiveness, robustness, and generality of the user attention model. The promising results from the user study on video summarization indicate that the user attention model is an alternative way to video understanding.