Learning to Predict Layout-to-image Conditional Convolutions for\n Semantic Image Synthesis

Abstract

Semantic image synthesis aims at generating photorealistic images from\nsemantic layouts. Previous approaches with conditional generative adversarial\nnetworks (GAN) show state-of-the-art performance on this task, which either\nfeed the semantic label maps as inputs to the generator, or use them to\nmodulate the activations in normalization layers via affine transformations. We\nargue that convolutional kernels in the generator should be aware of the\ndistinct semantic labels at different locations when generating images. In\norder to better exploit the semantic layout for the image generator, we propose\nto predict convolutional kernels conditioned on the semantic label map to\ngenerate the intermediate feature maps from the noise maps and eventually\ngenerate the images. Moreover, we propose a feature pyramid semantics-embedding\ndiscriminator, which is more effective in enhancing fine details and semantic\nalignments between the generated images and the input semantic layouts than\nprevious multi-scale discriminators. We achieve state-of-the-art results on\nboth quantitative metrics and subjective evaluation on various semantic\nsegmentation datasets, demonstrating the effectiveness of our approach.\n