Real-time combined 2D+3D active appearance models

TLDR

Active appearance models are generative face models that, like 3D morphable models, can represent 3D facial motion despite being 2D. The study examines the representational power of AAMs relative to 3DMMs and proposes a real‑time combined 2D+3D AAM fitting algorithm. Using non‑rigid structure‑from‑motion, the authors derive 3D shape modes from a 2D AAM and constrain the model to generate only instances realizable by those modes, enabling real‑time fitting. They find that 2D AAMs require more shape parameters than 3DMMs yet can generate model instances that 3DMMs cannot produce.

Abstract

Active appearance models (AAMs) are generative models commonly used to model faces. Another closely related types of face models are 3D morphable models (3DMMs). Although AAMs are 2D, they can still be used to model 3D phenomena such as faces moving across pose. We first study the representational power of AAMs and show that they can model anything a 3DMM can, but possibly require more shape parameters. We quantify the number of additional parameters required and show that 2D AAMs can generate model instances that are not possible with the equivalent 3DMM. We proceed to describe how a non-rigid structure-from-motion algorithm can be used to construct the corresponding 3D shape modes of a 2D AAM. We then show how the 3D modes can be used to constrain the AAM so that it can only generate model instances that can also be generated with the 3D modes. Finally, we propose a real-time algorithm for fitting the AAM while enforcing the constraints, creating what we call a "combined 2D+3D AAM".