High-Fidelity Face Sketch-To-Photo Synthesis Using Generative Adversarial Network

Abstract

Face sketch-photo synthesis has important usage in law enforcement and human authentication. Due to the sparse information (no color or texture), the abstraction level, the diversity of sketches, and the domain gap between sketch and photo, it is challenging to synthesize a photo-realistic photo from an input sketch. Moreover, the deficiency of data also restricts the synthesis performance. In this paper, we present a high-fidelity face sketch-photo synthesis method using Generation Adversarial Network (GAN). Our network adopts a deep residual U-Net as generator and a Patch-GAN with residual blocks as discriminator. We design effective loss functions by enforcing pixels, edges and high-level features of the produced face photos. Moreover, we augment the CUHK sketch dataset using an effective sampling method. With the improved GAN and augmented dataset, we achieve high-fidelity face photos. Qualitative and quantitative experiments demonstrate the approach outperforms other method. Further experiments with a sketch-based photo editing application also validate the performance of our method.