Photorealistic rendering for augmented reality using environment illumination

TLDR

Mixing 3D computer‑generated images with real scenes in augmented reality requires accurate geometry, surface properties, and real‑scene lighting to achieve photorealism, but conventional rendering techniques are computationally intensive and unsuitable for real‑time performance. This work explores an image‑based and hardware‑based approach to improve photorealism for rendering synthetic objects in augmented reality. It employs image‑based lighting, environment illumination maps, and a simple yet practical multi‑pass rendering framework to achieve real‑time photorealistic rendering.

Abstract

Mixing 3D computer-generated images with real-scene images seamlessly in augmented reality has many desirable and wide areas of applications such as entertainment, cinematography, design visualization and medical trainings. The challenging task is to make virtual objects blend harmoniously into the real scene and appear as if they are like real. Apart from constructing detailed geometric 3D model representation and obtaining accurate surface properties for virtual objects, adopting real scene lighting information to render virtual objects is another important factor to achieve photorealistic rendering. Such a factor not only improves visual complexity of virtual objects, but also determines the consistency of illumination between the virtual objects and the surrounding real objects in the scene. Conventional rendering techniques such as ray tracing, and radiosity require intensive computation and data preparation to solve the lighting transport equation. Hence, they are less practical for rendering virtual objects in augmented reality, which demands a real-time performance. This work explores an image-based and hardware-based approach to improve photorealism for rendering synthetic objects in augmented reality. It uses a recent technique of image-based lighting, environment illumination maps, and a simple yet practical multi-pass rendering framework for augmented reality.