NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

TLDR

Depth‑based human activity analysis has shown strong performance, yet existing RGB+D benchmarks suffer from limited training data, few class labels, restricted camera views, and a narrow subject pool. The authors aim to provide a large RGB+D action dataset and develop a recurrent neural network that captures long‑term body‑part correlations for improved classification. The dataset comprises 60 action classes from 40 subjects, and the authors design a recurrent neural network that models long‑term temporal correlations of body‑part features. Deep learning outperforms handcrafted features on cross‑subject and cross‑view tests, and the dataset facilitates future data‑intensive learning research in RGB+D action recognition.

Abstract

Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+Dbased action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art handcrafted features on the suggested cross-subject and cross-view evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.