Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human\n Action Recognition

Abstract

Conventional 3D convolutional neural networks (CNNs) are computationally\nexpensive, memory intensive, prone to overfitting, and most importantly, there\nis a need to improve their feature learning capabilities. To address these\nissues, we propose spatio-temporal short term Fourier transform (STFT) blocks,\na new class of convolutional blocks that can serve as an alternative to the 3D\nconvolutional layer and its variants in 3D CNNs. An STFT block consists of\nnon-trainable convolution layers that capture spatially and/or temporally local\nFourier information using a STFT kernel at multiple low frequency points,\nfollowed by a set of trainable linear weights for learning channel\ncorrelations. The STFT blocks significantly reduce the space-time complexity in\n3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8\ntimes less computational costs when compared to the state-of-the-art methods.\nFurthermore, their feature learning capabilities are significantly better than\nthe conventional 3D convolutional layer and its variants. Our extensive\nevaluation on seven action recognition datasets, including Something-something\nv1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate\nthat STFT blocks based 3D CNNs achieve on par or even better performance\ncompared to the state-of-the-art methods.\n