Augmented touch without visual obtrusion

Abstract

Visuo-haptic mixed reality consists of adding to a real scene the ability to see and touch virtual objects. It requires the use of see-through display technology for visually mixing real and virtual objects, and haptic devices for adding haptic interaction with the virtual objects. However, haptic devices tend to be bulky items that appear in the field of view of the user. In this work, we propose a novel mixed reality paradigm where it is possible to touch and see virtual objects in combination with a real scene, but without visual obtrusion produced by the haptic device. This mixed reality paradigm relies on the following three technical steps: tracking of the haptic device, visual deletion of the device from the real scene, and background completion using image-based models. We have developed a successful proof-of-concept implementation, where a user can touch virtual objects in the context of a real scene.