Digital Twin Robotic System With Continuous Learning for Grasp Detection in Variable Scenes

Abstract

With the emergence of digitalization technology, digital twin bridges the gap between physical and virtual worlds in industrial production with synchronization, reliability, and fidelity. The manufacturing process of complex products needs multiple working procedures, where novel industrial parts occur, causing scenes to be variable for robots to perceive and grasp. Due to the geometric difference among objects in various categories, it is significant to empower robotic systems with the capability of continuous learning for grasp detection in variable scenes. Therefore, a digital twin robotic system is proposed to realize bidirectional real-time data interaction and synchronization in the physical and virtual worlds. In this digital twin robotic system, a synthetic grasp detection dataset composed of industrial parts is built for an industrial grasping task. Besides, a novel deep learning method, adaptive spatial-awareness grasp network with a novel cc, is proposed to realize end-to-end 7-DoF grasp detection for industrial objects. In addition, a continuous learning strategy is proposed for 7-DoF grasp pose detection without catastrophic forgetting in variable scenes. Experiments in both virtual and physical worlds have demonstrated the effectiveness of our method for potential industrial implementation, and the average grasping success rate reaches 91% and 88% for novel objects in the virtual and physical worlds, respectively.