A deep learning approach for specular highlight removal from transmissive materials

Abstract

Abstract The appearance of specular highlights in images is one main factor affecting accurate material or object recognition tasks. Such an appearance has a misleading effect on the true gradient information found in transmissive material images. Certain methods use specular highlights as an intrinsic feature of transparency to detect transparent objects. However, this process reduces the robustness of methods in applications with opaque and shiny materials and in the classification of tasks among related features, such as transparency and translucency. Thus, correcting this artefact can enhance texture‐ or gradient‐based image and video analyses. However, the correction of small or large regions with specular highlights from transmissive materials, such as glass, plastic and water, is complex and ambiguous. These materials are sensitive to specular highlights and exhibit high degrees of reflection. In this study, we propose a deep learning framework to address the problem. A partial convolution‐based inpainting method is integrated with automatic semantic mask generation by using a simple adaptive binarization to detect highlight spots during training and inference. The proposed framework improves the learning process by capturing the semantic nature of specular highlights. Moreover, the framework eliminates the use of image‐mask pairs during inference and avoids predefined irregular random mask training. We qualitatively and quantitatively evaluate the proposed framework by using new and existing publicly available datasets that contain specular images. Experimental results show that our framework registers competitive performance and considerably reduces computational time.