Image and depth from a conventional camera with a coded aperture

TLDR

Conventional cameras produce blurred images for out‑of‑focus regions, and prior all‑focus or depth‑extracting systems required extra hardware and lower spatial resolution. We propose a simple modification to a conventional camera that enables simultaneous recovery of a high‑resolution image and depth sufficient for semi‑automatic layered depth representation. By inserting a patterned occluder (coded aperture) into the lens and designing the pattern using a depth discriminability criterion, we recover both depth and an all‑focus image from a single photograph via a statistical image model, then extract a layered depth map with optional user strokes. The resulting sharp image and layered depth map can be combined for applications such as automatic scene segmentation, post‑exposure refocusing, and re‑rendering the scene from alternate viewpoints.

Abstract

A conventional camera captures blurred versions of scene information away from the plane of focus. Camera systems have been proposed that allow for recording all-focus images, or for extracting depth, but to record both simultaneously has required more extensive hardware and reduced spatial resolution. We propose a simple modification to a conventional camera that allows for the simultaneous recovery of both (a) high resolution image information and (b) depth information adequate for semi-automatic extraction of a layered depth representation of the image. Our modification is to insert a patterned occluder within the aperture of the camera lens, creating a coded aperture. We introduce a criterion for depth discriminability which we use to design the preferred aperture pattern. Using a statistical model of images, we can recover both depth information and an all-focus image from single photographs taken with the modified camera. A layered depth map is then extracted, requiring user-drawn strokes to clarify layer assignments in some cases. The resulting sharp image and layered depth map can be combined for various photographic applications, including automatic scene segmentation, post-exposure refocusing, or re-rendering of the scene from an alternate viewpoint.