TextureGAN: Controlling Deep Image Synthesis with Texture Patches

TLDR

Prior image synthesis approaches allow control via sketch and color strokes, but this work is the first to explore texture-based control. The study investigates deep image synthesis guided by sketch, color, and texture, aiming to train a generative network that produces objects consistent with user-specified texture patches. Users can place texture patches on sketches at arbitrary locations and scales, and the generative network is trained with a local texture loss in addition to adversarial and content losses to honor these cues. Experiments demonstrate that the proposed algorithm generates plausible images faithful to user controls, and ablation studies show it produces more realistic results than adapting existing methods.

Abstract

In this paper, we investigate deep image synthesis guided by sketch, color, and texture. Previous image synthesis methods can be controlled by sketch and color strokes but we are the first to examine texture control. We allow a user to place a texture patch on a sketch at arbitrary locations and scales to control the desired output texture. Our generative network learns to synthesize objects consistent with these texture suggestions. To achieve this, we develop a local texture loss in addition to adversarial and content loss to train the generative network. We conduct experiments using sketches generated from real images and textures sampled from a separate texture database and results show that our proposed algorithm is able to generate plausible images that are faithful to user controls. Ablation studies show that our proposed pipeline can generate more realistic images than adapting existing methods directly.