MediaPipe: A Framework for Building Perception Pipelines

TLDR

Building applications that perceive the world around them is challenging, requiring selection of ML algorithms, prototype development, balancing resource consumption and quality, and mitigating problematic cases. The MediaPipe framework addresses all of these challenges. Developers can use MediaPipe to build prototypes by combining existing perception components, advance them to polished cross‑platform applications, and measure system performance and resource consumption on target platforms. These features enable developers to focus on algorithm or model development and use MediaPipe as an environment for iteratively improving their application with results reproducible across different devices and platforms. MediaPipe will be open‑sourced at https://github.com/google/mediapipe.

Abstract

Building applications that perceive the world around them is challenging. A developer needs to (a) select and develop corresponding machine learning algorithms and models, (b) build a series of prototypes and demos, (c) balance resource consumption against the quality of the solutions, and finally (d) identify and mitigate problematic cases. The MediaPipe framework addresses all of these challenges. A developer can use MediaPipe to build prototypes by combining existing perception components, to advance them to polished cross-platform applications and measure system performance and resource consumption on target platforms. We show that these features enable a developer to focus on the algorithm or model development and use MediaPipe as an environment for iteratively improving their application with results reproducible across different devices and platforms. MediaPipe will be open-sourced at https://github.com/google/mediapipe.