Multimodal tactile sensing fused with vision for dexterous robotic housekeeping

TLDR

Robots are increasingly integrated into daily life, yet existing perceptual technologies fall short in multisensory integration, rapid response, and sensitive perception needed for home tasks. This work develops a flexible tactile sensor based on thin‑film thermistors and a tactile‑visual fusion architecture to enable dexterous robotic grasping and smart housekeeping. The sensor captures pressure, temperature, thermal properties, texture, and slip, while the fusion architecture integrates these modalities from low‑level sensing to high‑level decision making. The system achieves ultrasensitive (0.05 mm s⁻¹) and ultrafast (4 ms) slip detection, enabling rapid slip‑feedback control and accurate object recognition, and successfully performs autonomous desktop cleaning and multi‑item sorting.

Abstract

Abstract As robots are increasingly participating in our daily lives, the quests to mimic human abilities have driven the advancements of robotic multimodal senses. However, current perceptual technologies still unsatisfied robotic needs for home tasks/environments, particularly facing great challenges in multisensory integration and fusion, rapid response capability, and highly sensitive perception. Here, we report a flexible tactile sensor utilizing thin-film thermistors to implement multimodal perceptions of pressure, temperature, matter thermal property, texture, and slippage. Notably, the tactile sensor is endowed with an ultrasensitive (0.05 mm/s) and ultrafast (4 ms) slip sensing that is indispensable for dexterous and reliable grasping control to avoid crushing fragile objects or dropping slippery objects. We further propose and develop a robotic tactile-visual fusion architecture that seamlessly encompasses multimodal sensations from the bottom level to robotic decision-making at the top level. A series of intelligent grasping strategies with rapid slip feedback control and a tactile-visual fusion recognition strategy ensure dexterous robotic grasping and accurate recognition of daily objects, handling various challenging tasks, for instance grabbing a paper cup containing liquid. Furthermore, we showcase a robotic desktop-cleaning task, the robot autonomously accomplishes multi-item sorting and cleaning desktop, demonstrating its promising potential for smart housekeeping.