Panoptic Studio: A Massively Multiview System for Social Interaction\n Capture

Abstract

We present an approach to capture the 3D motion of a group of people engaged\nin a social interaction. The core challenges in capturing social interactions\nare: (1) occlusion is functional and frequent; (2) subtle motion needs to be\nmeasured over a space large enough to host a social group; (3) human appearance\nand configuration variation is immense; and (4) attaching markers to the body\nmay prime the nature of interactions. The Panoptic Studio is a system organized\naround the thesis that social interactions should be measured through the\nintegration of perceptual analyses over a large variety of view points. We\npresent a modularized system designed around this principle, consisting of\nintegrated structural, hardware, and software innovations. The system takes, as\ninput, 480 synchronized video streams of multiple people engaged in social\nactivities, and produces, as output, the labeled time-varying 3D structure of\nanatomical landmarks on individuals in the space. Our algorithm is designed to\nfuse the "weak" perceptual processes in the large number of views by\nprogressively generating skeletal proposals from low-level appearance cues, and\na framework for temporal refinement is also presented by associating body parts\nto reconstructed dense 3D trajectory stream. Our system and method are the\nfirst in reconstructing full body motion of more than five people engaged in\nsocial interactions without using markers. We also empirically demonstrate the\nimpact of the number of views in achieving this goal.\n