Learning to reconstruct shape and spatially-varying reflectance from a single image

TLDR

Reconstructing shape and reflectance from images is highly under‑constrained and traditionally requires specialized hardware or strong shape/reflectance assumptions. The study shows that non‑Lambertian, spatially‑varying BRDFs and complex geometry can be recovered from a single RGB image under unknown illumination and flash. We train a deep neural network that regresses shape and reflectance from the image, using a large‑scale dataset of procedurally generated shapes and real‑world SVBRDFs, a cascade multi‑scale architecture, and an in‑network rendering layer to handle global illumination. The approach achieves state‑of‑the‑art inverse rendering on synthetic and real data.

Abstract

Reconstructing shape and reflectance properties from images is a highly under-constrained problem, and has previously been addressed by using specialized hardware to capture calibrated data or by assuming known (or highly constrained) shape or reflectance. In contrast, we demonstrate that we can recover non-Lambertian, spatially-varying BRDFs and complex geometry belonging to any arbitrary shape class, from a single RGB image captured under a combination of unknown environment illumination and flash lighting. We achieve this by training a deep neural network to regress shape and reflectance from the image. Our network is able to address this problem because of three novel contributions: first, we build a large-scale dataset of procedurally generated shapes and real-world complex SVBRDFs that approximate real world appearance well. Second, single image inverse rendering requires reasoning at multiple scales, and we propose a cascade network structure that allows this in a tractable manner. Finally, we incorporate an in-network rendering layer that aids the reconstruction task by handling global illumination effects that are important for real-world scenes. Together, these contributions allow us to tackle the entire inverse rendering problem in a holistic manner and produce state-of-the-art results on both synthetic and real data.