Learning task-oriented grasping for tool manipulation from simulated self-supervision

TLDR

Tool manipulation is essential for robots to accomplish complex tasks, requiring reasoning about task effects and appropriate grasping, yet most robotics research focuses on task‑agnostic grasping that optimizes only for grasp robustness. This work introduces the Task‑Oriented Grasping Network (TOG‑Net) to jointly optimize task‑oriented grasping of tools and the corresponding manipulation policy. TOG‑Net is trained via large‑scale simulated self‑supervision on procedurally generated tool objects, and its performance is evaluated through simulated and real‑world experiments on sweeping and hammering tasks. The model achieves 71.1 % task success on sweeping and 80.0 % on hammering.

Abstract

Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and, thus, properly grasping and manipulating the tool to achieve the task. Most work in robotics has focused on task-agnostic grasping, which optimizes for only grasp robustness without considering the subsequent manipulation tasks. In this article, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering.