Summarizing visual data using bidirectional similarity

TLDR

Summarizing image or video data into smaller sizes requires retaining as much visual information as possible while minimizing new artifacts, a challenge addressed by the re‑targeting problem. The authors introduce a bi‑directional similarity measure that quantitatively captures these two requirements and propose it as a principled basis for visual data summarization. They formulate summarization as an optimization problem maximizing the bi‑directional similarity between the summary and the original data. Experiments demonstrate effective image and video summarization, and the same framework is applicable to tasks such as cropping, completion, synthesis, collage, object removal, and photo reshuffling.

Abstract

We propose a principled approach to summarization of visual data (images or video) based on optimization of a well-defined similarity measure. The problem we consider is re-targeting (or summarization) of image/video data into smaller sizes. A good ldquovisual summaryrdquo should satisfy two properties: (1) it should contain as much as possible visual information from the input data; (2) it should introduce as few as possible new visual artifacts that were not in the input data (i.e., preserve visual coherence). We propose a bi-directional similarity measure which quantitatively captures these two requirements: Two signals S and T are considered visually similar if all patches of S (at multiple scales) are contained in T, and vice versa. The problem of summarization/re-targeting is posed as an optimization problem of this bi-directional similarity measure. We show summarization results for image and video data. We further show that the same approach can be used to address a variety of other problems, including automatic cropping, completion and synthesis of visual data, image collage, object removal, photo reshuffling and more.