SORNet: Spatial Object-Centric Representations for Sequential\n Manipulation

Abstract

Sequential manipulation tasks require a robot to perceive the state of an\nenvironment and plan a sequence of actions leading to a desired goal state. In\nsuch tasks, the ability to reason about spatial relations among object entities\nfrom raw sensor inputs is crucial in order to determine when a task has been\ncompleted and which actions can be executed. In this work, we propose SORNet\n(Spatial Object-Centric Representation Network), a framework for learning\nobject-centric representations from RGB images conditioned on a set of object\nqueries, represented as image patches called canonical object views. With only\na single canonical view per object and no annotation, SORNet generalizes\nzero-shot to object entities whose shape and texture are both unseen during\ntraining. We evaluate SORNet on various spatial reasoning tasks such as spatial\nrelation classification and relative direction regression in complex tabletop\nmanipulation scenarios and show that SORNet significantly outperforms baselines\nincluding state-of-the-art representation learning techniques. We also\ndemonstrate the application of the representation learned by SORNet on\nvisual-servoing and task planning for sequential manipulation on a real robot.\n