Identifying Corresponding Patches in SAR and Optical Images With a Pseudo-Siamese CNN

Abstract

In this letter, we propose a pseudo-siamese convolutional neural network\n(CNN) architecture that enables to solve the task of identifying corresponding\npatches in very-high-resolution (VHR) optical and synthetic aperture radar\n(SAR) remote sensing imagery. Using eight convolutional layers each in two\nparallel network streams, a fully connected layer for the fusion of the\nfeatures learned in each stream, and a loss function based on binary\ncross-entropy, we achieve a one-hot indication if two patches correspond or\nnot. The network is trained and tested on an automatically generated dataset\nthat is based on a deterministic alignment of SAR and optical imagery via\npreviously reconstructed and subsequently co-registered 3D point clouds. The\nsatellite images, from which the patches comprising our dataset are extracted,\nshow a complex urban scene containing many elevated objects (i.e. buildings),\nthus providing one of the most difficult experimental environments. The\nachieved results show that the network is able to predict corresponding patches\nwith high accuracy, thus indicating great potential for further development\ntowards a generalized multi-sensor key-point matching procedure. Index\nTerms-synthetic aperture radar (SAR), optical imagery, data fusion, deep\nlearning, convolutional neural networks (CNN), image matching, deep matching\n