Learning to Interpret Natural Language Navigation Instructions from Observations

TLDR

The ability to understand natural-language instructions is critical to building intelligent agents that interact with humans. The paper presents a system that learns to transform natural-language navigation instructions into executable formal plans. The system learns from human demonstrations without prior linguistic knowledge, using a realistic corpus of complex English navigation instructions, and is evaluated in three complex virtual indoor environments with many objects and landmarks. Using a learned lexicon and a supervised semantic parser, the system automatically learns to correctly interpret a reasonable fraction of the complex instructions in the corpus.

Abstract

The ability to understand natural-language instructions is critical to building intelligent agents that interact with humans. We present a system that learns to transform natural-language navigation instructions into executable formal plans. Given no prior linguistic knowledge, the system learns by simply observing how humans follow navigation instructions. The system is evaluated in three complex virtual indoor environments with numerous objects and landmarks. A previously collected realistic corpus of complex English navigation instructions for these environments is used for training and testing data. By using a learned lexicon to refine inferred plans and a supervised learner to induce a semantic parser, the system is able to automatically learnto correctly interpret a reasonable fraction of the complex instructions in this corpus.