Augmented reality and haptic interfaces for robot‐assisted surgery

TLDR

Teleoperated robot‑assisted minimally invasive surgical systems currently underutilize haptic feedback that could enhance performance and safety. An interoperable interface was developed that delivers haptic feedback and 3D visual overlays through a software framework comprising control modules, robot and image‑processing plug‑ins, and surface reconstruction. The interface proved feasible in artificial‑tissue tasks—palpation for hard lumps and surface tracing with virtual‑fixture constraints—showing that it facilitates rapid development and effective haptic feedback in teleoperation. © 2011 John Wiley & Sons, Ltd.

Abstract

Abstract Background Current teleoperated robot‐assisted minimally invasive surgical systems do not take full advantage of the potential performance enhancements offered by various forms of haptic feedback to the surgeon. Direct and graphical haptic feedback systems can be integrated with vision and robot control systems in order to provide haptic feedback to improve safety and tissue mechanical property identification. Methods An interoperable interface for teleoperated robot‐assisted minimally invasive surgery was developed to provide haptic feedback and augmented visual feedback using three‐dimensional (3D) graphical overlays. The software framework consists of control and command software, robot plug‐ins, image processing plug‐ins and 3D surface reconstructions. Results The feasibility of the interface was demonstrated in two tasks performed with artificial tissue: palpation to detect hard lumps and surface tracing, using vision‐based forbidden‐region virtual fixtures to prevent the patient‐side manipulator from entering unwanted regions of the workspace. Conclusions The interoperable interface enables fast development and successful implementation of effective haptic feedback methods in teleoperation. Copyright © 2011 John Wiley & Sons, Ltd.