Virtual reality control of a robotic camera holder for minimally invasive surgery

Abstract

This paper presents a virtual reality(VR) control system for a robotic endoscope holder for minimally invasive surgery (MIS). The system features a three degree of freedom(DOF) robotic endoscope holder, virtual reality head mounted display and a control system that allows robot to follow headset movement. The headset display is aligned with endoscope camera view thus creating a full presence effect for a surgeon. The video from endoscope is streamed to the virtual reality head mounted display after image processing in a computer. During surgery, the proposed control system generates commands to the robot based on the headset's positioning data. After the desired camera motion direction is estimated, the robotic endoscope holder receives motion commands through serial interface. The proposed virtual reality robot control system is implemented in Unity. The system has been tested in the laboratory environment through a set of motion tests and achieved a response latency of 2s. The proposed system is able to restore the surgeon's perceptual capability of the operation space that is highly restricted in MIS and provide highly intuitive endoscope robot control.