2023 · 52 citations · 27 references
ComputingVisualizationComputer VisionPortraitureRendering3D Animation3D RenderingAnimation RenderingAvatar DesignImage SynthesisAccuracy-computational Cost TradeoffAvatar Facial ExpressionIdentityMachine Learning-based InversionNeural Field ModelsNeural Field TheoryOptimization-based InversionSubject-specific Effects
Controllability, generalizability and efficiency are the major objectives of constructing face avatars represented by neural implicit field. However, existing methods have not managed to accommodate the three requirements simultaneously. They either focus on static portraits, restricting the representation ability to a specific subject, or suffer from substantial computational cost, limiting their flexibility. In this paper, we propose One-shot Talking face Avatar (OTAvatar), which constructs face avatars by a generalized controllable tri-plane rendering solution so that each personalized avatar can be constructed from only one portrait as the reference. Specifically, OTAvatar first inverts a portrait image to a motion-free identity code. Second, the identity code and a motion code are utilized to modulate an efficient CNN to generate a tri-plane formulated volume, which encodes the subject in the desired motion. Finally, volume rendering is employed to generate an image in any view. The core of our solution is a novel decoupling-by-inverting strategy that disentangles identity and motion in the latent code via optimization-based inversion. Benefiting from the efficient tri-plane representation, we achieve controllable rendering of generalized face avatar at 35 FPS on AIOO. Experiments show promising performance of crossidentity reenactment on subjects out of the training set and better 3D consistency. The code is available at https://github.com/theEricMaIOTAvatar.
27
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh et al. · IEEE Transactions on Image Processing · 2004 · 56K citations
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros et al. · 2018 · 12.4K citations
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik et al. · Communications of the ACM · 2021 · 5.8K citations · Full text
pixelNeRF: Neural Radiance Fields from One or Few Images
Alex Yu, Vickie Ye, Matthew Tancik et al. · 2021 · 1.4K citations