Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking

Abstract

Thanks to the abundant spectral bands, hyperspectral videos (HSVs) are able to describe objects at material level, i.e., the physical property, providing more benefits for object tracking than color videos. Considering limited HSV dataset for training, a band attention aware ensemble network was recently proposed for hyperspectral tracking, which leverages band attention to select several three-channel images for deep hyperspectral tracking. However, it fails to fully consider the joint spectral-spatial-temporal information in HSVs, compromising its tracking performance in challenging scenarios. To this end, we introduce a spectral-spatial-temporal attention neural network (SST-Net) for hyperspectral tracking in this paper. Specifically, the spatial attention with convolution and deconvolution structure focuses on the salient spatial features. Moreover, the temporal attention with an RNN structure is adopted to depict the temporal relationship among adjacent frames. By combining the spatial, spectral, and temporal attention, the band relationship can be better depicted thus valuable hyperspectral bands can be better selected for deep ensemble tracking. Experimental results show the improved effectiveness of SST-Net in tracking over serval alternative trackers.