3D Reconstruction of “In-the-Wild” Faces in Images and Videos

James G. Booth, Anastasios Roussos, Evangelos Ververas, Epameinondas Antonakos, Stylianos Ploumpis, Yannis Panagakis, Stefanos Zafeiriou

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2018 · 62 citations · 70 references

Concepts

TL;DR

3D Morphable Models are state‑of‑the‑art for reconstructing facial shape, but existing datasets are captured under controlled conditions, limiting texture models for in‑the‑wild images. This work proposes the first in‑the‑wild 3D Morphable Model that integrates a statistical identity‑expression shape model with an in‑the‑wild texture model. The model is built by combining a statistical shape model with an in‑the‑wild texture model, enabling a simplified fitting procedure that does not require illumination optimization. On three newly collected in‑the‑wild benchmarks, the approach achieves state‑of‑the‑art reconstruction accuracy.

Abstract

3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and are among the state-of-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions ("in-the-wild"). In this paper, we propose the first "in-the-wild" 3DMM by combining a statistical model of facial identity and expression shape with an "in-the-wild" texture model. We show that such an approach allows for the development of a greatly simplified fitting procedure for images and videos, as there is no need to optimise with regards to the illumination parameters. We have collected three new benchmarks that combine "in-the-wild" images and video with ground truth 3D facial geometry, the first of their kind, and report extensive quantitative evaluations using them that demonstrate our method is state-of-the-art.

References

70