A Joint Model of Language and Perception for Grounded Attribute Learning

TLDR

Robots are increasingly ubiquitous, making it essential for untrained users to interact easily, which has spurred research into language grounding to link natural language meanings to the physical world. The study proposes a joint learning approach for language and perception models to induce grounded attributes. The perception component uses classifiers for physical characteristics, while the language model employs a probabilistic categorial grammar to build compositional meaning representations. Evaluation on interpreting object‑set descriptions in a physical workspace shows accurate task performance and effective latent‑variable concept induction.

Abstract

As robots become more ubiquitous and capable, it becomes ever more important for untrained users to easily interact with them. Recently, this has led to study of the language grounding problem, where the goal is to extract representations of the meanings of natural language tied to the physical world. We present an approach for joint learning of language and perception models for grounded attribute induction. The perception model includes classifiers for physical characteristics and a language model based on a probabilistic categorial grammar that enables the construction of compositional meaning representations. We evaluate on the task of interpreting sentences that describe sets of objects in a physical workspace, and demonstrate accurate task performance and effective latent-variable concept induction in physical grounded scenes.