Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

TLDR

Robots must comprehend spoken natural language to communicate with humans, yet unconstrained spoken instructions pose challenges due to complex structures, diverse expressions, and inherent ambiguity. The paper proposes the first comprehensive system for controlling robots with unconstrained spoken language that effectively resolves instruction ambiguity. The system integrates deep learning‑based object detection with natural language processing and uses dialogue to resolve instruction ambiguity. Experiments on a simulated environment and a physical industrial robot arm show that the system effectively understands natural instructions and achieves higher object‑picking success rates via interactive clarification.

Abstract

Comprehension of spoken natural language is an essential skill for robots to communicate with humans effectively. However, handling unconstrained spoken instructions is challenging due to (1) complex structures and the wide variety of expressions used in spoken language, and (2) inherent ambiguity of human instructions. In this paper, we propose the first comprehensive system for controlling robots with unconstrained spoken language, which is able to effectively resolve ambiguity in spoken instructions. Specifically, we integrate deep learning-based object detection together with natural language processing technologies to handle unconstrained spoken instructions, and propose a method for robots to resolve instruction ambiguity through dialogue. Through our experiments on both a simulated environment as well as a physical industrial robot arm, we demonstrate the ability of our system to understand natural instructions from human operators effectively, and show how higher success rates of the object picking task can be achieved through an interactive clarification process.