Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors

Abstract

We present a demo of low-cost mixed reality telepresence system that performs real-time 3D reconstruction of a person or an object and wirelessly transmits the reconstructions to Microsoft's HoloLens head mounted display at frame rates perceived as smooth. A reconstructed frame is represented as a polygonal mesh with polygons textured with high definition data obtained from RGB cameras. Each frame is compressed and sent to HoloLens, so that it can be locally rendered by its GPU, minimizing latency of reacting to head movement. Owing to HoloLens half-translucent displays, the system creates an appearance of remote object being part of user's physical environment. The system can run on a relatively low-cost commodity hardware, such as Kinect sensors and in the most basic scenario can produce smooth frame rates of a single multicore laptop.