A Multi-Sensor Approach for Digital Twins of Manual Assembly and Commissioning

Abstract

Digital twins enable better control and provide a better understanding of automated manufacturing environments. However, some parts of production processes are not fully covered, as they are still carried out manually. In this paper, we present an approach for monitoring manual parts of an assembly and commissioning process efficiently. We combine scenery classification with an inventory control system based on high-precision scales. We use a head-mounted mixed-reality display to acquire images of the process context and to provide assistance to the worker. Hence, heterogeneous sensor information is combined to capture human activities and object interactions to create an accurate digital representation of actual process behavior. Additionally, sensor information is selected dynamically based on the context. The system was implemented and tested in a laboratory environment. For production environments, this provides a starting point to close a significant gap regarding digital twins in semi-automated manufacturing processes and enables context-aware worker assistance.