OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

TLDR

Realtime multi‑person 2D pose estimation is essential for machine perception of people in images and videos, and earlier work refined Part Affinity Fields and body part locations together during training. The authors aim to provide a realtime approach for detecting 2D poses of multiple people and introduce the first combined body‑and‑foot keypoint detector using a newly released foot dataset. They employ a bottom‑up, nonparametric Part Affinity Fields representation to associate body parts with individuals, and extend it with a combined body‑and‑foot detector trained on an internal foot dataset. The system achieves high accuracy and realtime performance regardless of crowd size, improves runtime and accuracy through PAF‑only refinement, and the combined detector reduces inference time while preserving component accuracy, culminating in the open‑source OpenPose release.

Abstract

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.