Edge Assisted Real-time Object Detection for Mobile Augmented Reality

TLDR

Most AR/MR systems capture 3D geometry but cannot detect and classify complex real‑world objects, and while deep CNNs could enable this, running large networks on mobile or offloading to edge/cloud is difficult due to stringent accuracy and latency requirements that degrade with user view changes. To address this, we design a system that enables high‑accuracy object detection for commodity AR/MR devices running at 60 fps. The system employs low‑latency offloading, decouples the rendering pipeline from the offloading pipeline, and applies a fast object‑tracking method to preserve detection accuracy. The system improves detection accuracy by 20.2–34.8 % for object detection and human keypoint detection, requires only 2.24 ms latency for tracking, and frees resources for higher‑quality AR/MR rendering.

Abstract

Most existing Augmented Reality (AR) and Mixed Reality (MR) systems are able to understand the 3D geometry of the surroundings but lack the ability to detect and classify complex objects in the real world. Such capabilities can be enabled with deep Convolutional Neural Networks (CNN), but it remains difficult to execute large networks on mobile devices. Offloading object detection to the edge or cloud is also very challenging due to the stringent requirements on high detection accuracy and low end-to-end latency. The long latency of existing offloading techniques can significantly reduce the detection accuracy due to changes in the user's view. To address the problem, we design a system that enables high accuracy object detection for commodity AR/MR system running at 60fps. The system employs low latency offloading techniques, decouples the rendering pipeline from the offloading pipeline, and uses a fast object tracking method to maintain detection accuracy. The result shows that the system can improve the detection accuracy by 20.2%-34.8% for the object detection and human keypoint detection tasks, and only requires 2.24ms latency for object tracking on the AR device. Thus, the system leaves more time and computational resources to render virtual elements for the next frame and enables higher quality AR/MR experiences.