Neural Reflectance Fields for Appearance Acquisition

TLDR

The paper introduces Neural Reflectance Fields, a deep scene representation that encodes volume density, normals, and reflectance at any 3D point using a fully‑connected neural network. The authors employ a physically‑based differentiable ray‑marching framework to render images from the neural reflectance field and estimate it from images captured with a simple collocated camera‑light setup. Neural reflectance fields can be estimated from such simple images, accurately reproduce complex real‑world appearance, enable photo‑realistic novel‑view and relighting rendering that outperforms prior methods, and can be composed with traditional scene models for a complete 3D rendering pipeline.

Abstract

We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene using a fully-connected neural network. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light. We demonstrate that neural reflectance fields can be estimated from images captured with a simple collocated camera-light setup, and accurately model the appearance of real-world scenes with complex geometry and reflectance. Once estimated, they can be used to render photo-realistic images under novel viewpoint and (non-collocated) lighting conditions and accurately reproduce challenging effects like specularities, shadows and occlusions. This allows us to perform high-quality view synthesis and relighting that is significantly better than previous methods. We also demonstrate that we can compose the estimated neural reflectance field of a real scene with traditional scene models and render them using standard Monte Carlo rendering engines. Our work thus enables a complete pipeline from high-quality and practical appearance acquisition to 3D scene composition and rendering.