Communicating and controlling robot arm motion intent through mixed-reality head-mounted displays

TLDR

Efficient communication of robot motion intent is essential for safe human–robot collaboration, yet robots struggle to use human non‑verbal cues and current systems rely on 2D displays that force workers to pause their tasks. The study proposes a mixed‑reality HMD that overlays intended robot motion onto the user’s real‑world view. The system connects a ROS‑enabled robot to a HoloLens via ROS Reality, uses MoveIt for motion planning and Unity for rendering, lets users adjust the end‑effector goal pose with hand gestures, and was evaluated with 32 participants labeling arm trajectories for collision. Participants achieved a 15 % higher accuracy and 38 % faster task completion using the HMD compared with the best baseline, confirming that mixed‑reality visualization improves human assessment of robot motion.

Abstract

Efficient motion intent communication is necessary for safe and collaborative work environments with co-located humans and robots. Humans efficiently communicate their motion intent to other humans through gestures, gaze, and other non-verbal cues, and can replan their motions in response. However, robots often have difficulty using these methods. Many existing methods for robot motion intent communication rely on 2D displays, which require the human to continually pause their work to check a visualization. We propose a mixed-reality head-mounted display (HMD) visualization of the intended robot motion over the wearer’s real-world view of the robot and its environment. In addition, our interface allows users to adjust the intended goal pose of the end effector using hand gestures. We describe its implementation, which connects a ROS-enabled robot to the HoloLens using ROS Reality, using MoveIt for motion planning, and using Unity to render the visualization. To evaluate the effectiveness of this system against a 2D display visualization and against no visualization, we asked 32 participants to label various arm trajectories as either colliding or non-colliding with blocks arranged on a table. We found a 15% increase in accuracy with a 38% decrease in the time it took to complete the task compared with the next best system. These results demonstrate that a mixed-reality HMD allows a human to determine where the robot is going to move more quickly and accurately than existing baselines.