Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

TLDR

Human3.6M is a new dataset of 3.6 million accurate 3‑D human poses collected from 11 subjects across four viewpoints, designed to train realistic human‑sensing systems and benchmark future pose‑estimation models. The dataset supplies synchronized RGB, depth, motion‑capture, and 3‑D body‑scan data, along with mixed‑reality evaluation scenarios that insert animated models into complex real environments under moving cameras and occlusion, and provides large‑scale statistical models and evaluation baselines. Experiments show that a large‑scale model trained on Human3.6M outperforms the largest existing public dataset by 20 %, and the dataset and accompanying code are publicly available online.

Abstract

We introduce a new dataset, Human3.6M, of 3.6 Million accurate 3D Human poses, acquired by recording the performance of 5 female and 6 male subjects, under 4 different viewpoints, for training realistic human sensing systems and for evaluating the next generation of human pose estimation models and algorithms. ), with additional synchronized image, human motion capture, and time of flight (depth) data, and with accurate 3D body scans of all the subject actors involved. We also provide controlled mixed reality evaluation scenarios where 3D human models are animated using motion capture and inserted using correct 3D geometry, in complex real environments, viewed with moving cameras, and under occlusion. Finally, we provide a set of large-scale statistical models and detailed evaluation baselines for the dataset illustrating its diversity and the scope for improvement by future work in the research community. Our experiments show that our best large-scale model can leverage our full training set to obtain a 20% improvement in performance compared to a training set of the scale of the largest existing public dataset for this problem. Yet the potential for improvement by leveraging higher capacity, more complex models with our large dataset, is substantially vaster and should stimulate future research. The dataset together with code for the associated large-scale learning models, features, visualization tools, as well as the evaluation server, is available online at http://vision.imar.ro/human3.6m.