Exemplar Guided Unsupervised Image-to-Image Translation with Semantic\n Consistency

Abstract

Image-to-image translation has recently received significant attention due to\nadvances in deep learning. Most works focus on learning either a one-to-one\nmapping in an unsupervised way or a many-to-many mapping in a supervised way.\nHowever, a more practical setting is many-to-many mapping in an unsupervised\nway, which is harder due to the lack of supervision and the complex inner- and\ncross-domain variations. To alleviate these issues, we propose the Exemplar\nGuided & Semantically Consistent Image-to-image Translation (EGSC-IT) network\nwhich conditions the translation process on an exemplar image in the target\ndomain. We assume that an image comprises of a content component which is\nshared across domains, and a style component specific to each domain. Under the\nguidance of an exemplar from the target domain we apply Adaptive Instance\nNormalization to the shared content component, which allows us to transfer the\nstyle information of the target domain to the source domain. To avoid semantic\ninconsistencies during translation that naturally appear due to the large\ninner- and cross-domain variations, we introduce the concept of feature masks\nthat provide coarse semantic guidance without requiring the use of any semantic\nlabels. Experimental results on various datasets show that EGSC-IT does not\nonly translate the source image to diverse instances in the target domain, but\nalso preserves the semantic consistency during the process.\n