Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set

TLDR

Deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency, but require large amounts of data while ground‑truth 3D face shapes are scarce. We propose a novel deep 3D face reconstruction approach that uses a robust hybrid loss for weakly‑supervised learning and performs multi‑image reconstruction by aggregating complementary information. The method employs a hybrid loss combining low‑level and perception‑level cues and aggregates shape information across multiple images to improve reconstruction. Our experiments on MICC Florence and Facewarehouse datasets demonstrate that the method is fast, accurate, robust to occlusion and large pose, and outperforms fifteen recent state‑of‑the‑art methods. Code is available at https://github.com/Microsoft/Deep3DFaceReconstruction.

Abstract

Recently, deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency. However, training deep neural networks typically requires a large volume of data, whereas face images with ground-truth 3D face shapes are scarce. In this paper, we propose a novel deep 3D face reconstruction approach that 1) leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and 2) performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation. Our method is fast, accurate, and robust to occlusion and large pose. We provide comprehensive experiments on MICC Florence and Facewarehouse datasets, systematically comparing our method with fifteen recent methods and demonstrating its state-of-the-art performance. Code available at https://github.com/Microsoft/Deep3DFaceReconstruction.