Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks

TLDR

Skeleton-based action recognition uses joint trajectories as a strong representation for describing human actions. The study proposes an end‑to‑end fully connected deep LSTM network that learns co‑occurrence features of skeleton joints for action recognition. The network incorporates a novel regularization scheme to capture joint co‑occurrences and a dropout algorithm applied simultaneously to gates, cells, and outputs to train the deep LSTM effectively. Experimental results on three datasets consistently demonstrate the effectiveness of the proposed model.

Abstract

Skeleton based action recognition distinguishes human actions using the trajectories of skeleton joints, which provide a very good representation for describing actions. Considering that recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) can learn feature representations and model long-term temporal dependencies automatically, we propose an end-to-end fully connected deep LSTM network for skeleton based action recognition. Inspired by the observation that the co-occurrences of the joints intrinsically characterize human actions, we take the skeleton as the input at each time slot and introduce a novel regularization scheme to learn the co-occurrence features of skeleton joints. To train the deep LSTM network effectively, we propose a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons. Experimental results on three human action recognition datasets consistently demonstrate the effectiveness of the proposed model.