A Multi-View Human Action Recognition System in Limited Data Case using Multi-Stream CNN

Abstract

In recent years, Convolutional Neural Networks (CNNs) have been extensively used for human action recognition. However, training a convolutional neural network by limited data is a challenging problem. In this paper, a multi-stream 3D-CNN structure is proposed for multi-view human action recognition. In this model, a four-stream 3D-CNN with handcrafted features, containing optical flow and gradients in horizontal and vertical directions, is proposed as a solution to improve the recognition performance in the case of limited data. This model combines multi-view four-stream 3D-CNNs from different views. The proposed multi-stream 3D-CNN is applied to IXMAS and NIXMAS multi-view datasets. The experiments illustrate superior results in comparison with state-of-the-art methods. The results show 3.58% improvement in comparison with single stream 3D-CNN architecture using raw video data in IXMAS dataset. However, with more limitations in number of training data in NIXMAS dataset, results show remarkable improvement in comparison with single stream 3D-CNN structure that is 12.6%.