3D-Aided Deep Pose-Invariant Face Recognition

TLDR

Learning from synthetic faces, while data‑efficient, often fails to match real‑world performance due to distribution gaps between synthetic and real images. To bridge this gap, we propose 3D‑Aided Deep Pose‑Invariant Face Recognition (3D‑PIM), which automatically reconstructs realistic frontal faces from arbitrary poses using a 3D face model. 3D‑PIM first employs a 3D Morphable Model–based simulator to generate shape and appearance priors, then refines the simulated faces with a global‑local GAN trained on unlabelled real data to preserve identity and enhance realism. Experiments on controlled and in‑the‑wild benchmarks show that 3D‑PIM outperforms state‑of‑the‑art pose‑invariant face recognition methods.

Abstract

Learning from synthetic faces, though perhaps appealing for high data efficiency, may not bring satisfactory performance due to the distribution discrepancy of the synthetic and real face images. To mitigate this gap, we propose a 3D-Aided Deep Pose-Invariant Face Recognition Model (3D-PIM), which automatically recovers realistic frontal faces from arbitrary poses through a 3D face model in a novel way. Specifically, 3D-PIM incorporates a simulator with the aid of a 3D Morphable Model (3D MM) to obtain shape and appearance prior for accelerating face normalization learning, requiring less training data. It further leverages a global-local Generative Adversarial Network (GAN) with multiple critical improvements as a refiner to enhance the realism of both global structures and local details of the face simulator’s output using unlabelled real data only, while preserving the identity information. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks clearly demonstrate superiority of the proposed model over state-of-the-arts.