Mental Imagery for a Conversational Robot

TLDR

Language links to vision by encoding points of view, implying visual perspective shifts and relying on object permanence, allowing us to refer to unseen objects. The study aims to equip robots with the ability to link speech to vision by imagining perspective shifts and maintaining object permanence, enabling fluid face‑to‑face conversations. The authors develop representations and procedures that couple active vision with physical simulation to build a robotic manipulator’s mental model of its environment, enabling mental imagery for spatial language. The model allows generation of imagined views from arbitrary perspectives, supporting situated language comprehension and production, as demonstrated in an initial spatial language application.

Abstract

To build robots that engage in fluid face-to-face spoken conversations with people, robots must have ways to connect what they say to what they see. A critical aspect of how language connects to vision is that language encodes points of view. The meaning of my left and your left differs due to an implied shift of visual perspective. The connection of language to vision also relies on object permanence. We can talk about things that are not in view. For a robot to participate in situated spoken dialog, it must have the capacity to imagine shifts of perspective, and it must maintain object permanence. We present a set of representations and procedures that enable a robotic manipulator to maintain a "mental model" of its physical environment by coupling active vision to physical simulation. Within this model, "imagined" views can be generated from arbitrary perspectives, providing the basis for situated language comprehension and production. An initial application of mental imagery for spatial language understanding for an interactive robot is described.