A Novel Method of Combining Computer Vision, Eye-Tracking, EMG, and IMU to Control Dexterous Prosthetic Hand

Abstract

Due to poor robustness, instability, and the heavy burden of use, the traditional myoelectric control method is still powerless in the face of the control of the dexterous prosthetic hand. To solve this problem, a new method (CVEEI), that combines computer vision, eye tracking, electromyogram (EMG) and IMU was proposed in this paper. Firstly, through gazing (eye-tracking) in front of the screen, the grasping pattern of the objects can be fed back to the prosthetic hand controller; Then, the prosthetic hand can be controlled to transport the object to the position expected, on collaboration of both EMG and IMU. In this process, the grasping pattern of all objects can be recognized by computer vision in real-time. Importantly, through comparing the traditional EMG control method (co-contraction to switch) in the transport experiment of the objects, the superiority of this new method in operating the dexterous prosthetic hand was further verified (fast > 1s/single object) in this paper.