3D tracking of facial features for augmented reality applications

Abstract

We present an algorithm for feature-based real-time 3D tracking of facial features and its application for visualization of virtual facial modifications. A non-linear Kalman-based estimator is used for 3D head pose calculation and accurate facial features localization. The 3D face model used is adapted to the particular user’s face by utilizing active shape model for facial landmarks detection, followed by z-depth progressive refining. Virtual facial modifications are performed by user-driven 3D-aware 2D warping of the image sequence. The evaluation of our system shows that the tracker is robust to moderate head movements, occlusion and facial animation, which results in quite realistically-looking virtual facial modifications.