Rendering synthetic objects into real scenes

TLDR

The paper proposes a method that uses measured scene radiance and global illumination to add synthetic objects into real scenes with correct lighting. It employs an HDR image‑based scene model split into distant, local, and synthetic components, assumes the distant part is photometrically unchanged, estimates reflectance for the local part to capture shadows and reflections, and applies global illumination to simulate light interactions. Differential rendering yields good results even when only an estimate of the local scene’s reflectance is available.

Abstract

We present a method that uses measured scene radiance and global illumination in order to add new objects to light-based models with correct lighting. The method uses a high dynamic range image-based model of the scene, rather than synthetic light sources, to illuminate the new objects. To compute the illumination, the scene is considered as three components: the distant scene, the local scene, and the synthetic objects. The distant scene is assumed to be photometrically unaffected by the objects, obviating the need for reflectance model information. The local scene is endowed with estimated reflectance model information so that it can catch shadows and receive reflected light from the new objects. Renderings are created with a standard global illumination method by simulating the interaction of light amongst the three components. A differential rendering technique allows for good results to be obtained when only an estimate of the local scene reflectance properties is known.