Self-Driven Dual-Path Learning for Reference-Based Line Art Colorization Under Limited Data

Abstract

Synthesizing color images based on line arts while considering the styles of reference photos is a flexible form of artistic creation that has recently attracted public attention. Previous approaches usually require large datasets at training, causing great inconvenience to the application. Besides, the sparsity of line art pictures often leads to a failure in learning valid mappings. To this end, we present SDL, a self-driven dual-path framework for reference-based line art colorization under limited data. Given small training sets containing sketch-image pairs, SDL first utilizes a novel Dynamic Pseudo Sample Generator (DPSG) to produce quantities of fake samples. Then, we introduce a dual-path network to achieve better visual effects, in which the Content-Generation Path reconstructs reliable content features to help establish multi-level correspondence in the Content-Color Aggregation Module (CCAM) of the Color-Transfer Path. Furthermore, we develop a Region-aware Contrastive Scheme (RCS) to focus on fine-grained details and a Style-augmented Contrastive Scheme (SCS) to encourage style consistency. Experiments verify the superiority of our model compared with existing works. We also demonstrate SDL outperforms state-of-the-art self-driven methods even though they adopt much more data than us ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$30\times $ </tex-math></inline-formula> on CelebA-HQ Dataset and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$17\times $ </tex-math></inline-formula> on ASCP Dataset).