Accurate Passive 3D Polarization Face Reconstruction under Complex Conditions Assisted with Deep Learning

Abstract

Accurate passive 3D face reconstruction is of great importance with various potential applications. Three-dimensional polarization face reconstruction is a promising approach, but one bothered by serious deformations caused by an ambiguous surface normal. In this study, we propose a learning-based method for passive 3D polarization face reconstruction. It first calculates the surface normal of each microfacet at a pixel level based on the polarization of diffusely reflected light on the face, where no auxiliary equipment, including artificial illumination, is required. Then, the CNN-based 3DMM (convolutional neural network; 3D morphable model) generates a rough depth map of the face with the directly captured polarization image. The map works as an extra constraint to correct the ambiguous surface normal obtained from polarization. An accurate surface normal finally allows for an accurate 3D face reconstruction. Experiments in both indoor and outdoor conditions demonstrate that accurate 3D faces can be well-reconstructed. Moreover, with no auxiliary equipment required, the method ensures a total passive 3D face reconstruction.