Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

TLDR

The study combines generative Markov random field models with discriminatively trained deep convolutional neural networks to synthesize 2D images. The method applies a generative MRF to higher‑level layers of a dCNN feature pyramid to control image layout, and is used for both photographic and non‑photo‑realistic synthesis tasks. The combined MRF–dCNN system reduces over‑excitation artifacts and implausible feature mixtures, producing more visually plausible photographic images and achieving greater variability than classic MRF methods.

Abstract

This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN feature pyramid, controlling the image layout at an abstract level. We apply the method to both photographic and non-photo-realistic (artwork) synthesis tasks. The MRF regularizer prevents over-excitation artifacts and reduces implausible feature mixtures common to previous dCNN inversion approaches, permitting synthesizing photographic content with increased visual plausibility. Unlike standard MRF-based texture synthesis, the combined system can both match and adapt local features with considerable variability, yielding results far out of reach of classic generative MRF methods.