Learning to Follow Navigational Directions

TLDR

Aligning natural language directions with actual routes is challenging because there is no explicit mapping between text and path segments. The study introduces a system that learns to follow navigational natural language directions. The system learns by apprenticeship from map routes paired with English descriptions, using reinforcement learning that rewards staying close to the intended path. The system successfully grounds spatial terms such as above and south into geometric path properties.

Abstract

We present a system that learns to follow navigational natural language directions. Where traditional models learn from linguistic annotation or word distributions, our approach is grounded in the world, learning by apprenticeship from routes through a map paired with English descriptions. Lacking an explicit alignment between the text and the reference path makes it difficult to determine what portions of the language describe which aspects of the route. We learn this correspondence with a reinforcement learning algorithm, using the deviation of the route we follow from the intended path as a reward signal. We demonstrate that our system successfully grounds the meaning of spatial terms like above and south into geometric properties of paths.