PaLM-E: An Embodied Multimodal Language Model

TLDR

Large language models excel at a wide range of complex tasks, but grounding them for real‑world inference, such as robotics, remains a challenge. We propose embodied language models that directly incorporate continuous sensor modalities into language models to link words with percepts. The model interleaves visual, continuous state estimation, and textual encodings, trained end‑to‑end with a pre‑trained LLM for tasks like robotic manipulation planning, visual question answering, and captioning. PaLM‑E, a single large embodied multimodal model, handles diverse embodied reasoning tasks across modalities and embodiments, shows positive transfer from joint training, and its largest variant PaLM‑E‑562B achieves state‑of‑the‑art OK‑VQA performance while retaining generalist language abilities.

Abstract

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.