Scene completion using millions of photographs

TLDR

A million images can be leveraged to show that although the image space is vast, the space of semantically distinct scenes is relatively small. This paper introduces a new image‑completion algorithm that uses a massive web‑gathered photo database. The algorithm patches image holes by locating seamless, semantically valid regions in the database, and it is entirely data‑driven without requiring user annotations. The method successfully completes many images, provides diverse selectable results, and outperforms existing image‑completion approaches.

Abstract

What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of results for each input image and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches.