Deep Learning-Based Spatiotemporal Fusion of Unmanned Aerial Vehicle and Satellite Reflectance Images for Crop Monitoring

Abstract

Spatiotemporal fusion (STF) techniques play important roles in Earth observation analysis as they can produce images with both high spatial and temporal resolution. However, existing STF models often fuse images from various satellites, not satisfying the demand for precise crop monitoring. In contrast, unmanned aerial vehicle (UAV) images can deliver detailed data, and deep learning (DL)-based STF models have a high potential for automatically extracting abstract features. To this end, this study proposed a novel end-to-end DL-based STF model named UAV-Net, which can produce centimeter-scale UAV images. UAV-Net has an encoder-decoder architecture with Modified ResNet (MResNet), Feature Pyramid Network (FPN), and decoder modules. The encoder uses MResNet modules to extract input features, while the FPN module performs a multiscale fusion of these features before reconstructing UAV images using transposed convolution in the decoder module. Through the comparison and ablation experiments, this study evaluated the efficacies of the MResNet modules with 18, 34, and 50 layers, along with FPN module of UAV-Net. The experimental results on real-world datasets demonstrated that UAV-Net adequately produce UAV images both visually and quantitatively. Furthermore, Comparison with state-of-the-art STF models highlights the innovative and effective of UAV-Net for producing centimeter-scale images. The predicted centimeter-scale images have the potential to be useful for various environmental monitoring applications.