Haptic perception using optoelectronic robotic flesh for embodied artificially intelligent agents

Flesh encodes a variety of haptic information including deformation, temperature, vibration, and damage stimuli using a multisensory array of mechanoreceptors distributed on the surface of the human body. Currently, soft sensors are capable of detecting some haptic stimuli, but whole-body multimodal perception at scales similar to a human adult (surface area ~17,000 square centimeters) is still a challenge in artificially intelligent agents due to the lack of encoding. This encoding is needed to reduce the wiring required to send the vast amount of information transmitted to the processor. We created a robotic flesh that could be further developed for use in these agents. This engineered flesh is an optical, elastomeric matrix “innervated” with stretchable lightguides that encodes haptic stimuli into light: temperature into wavelength due to thermochromic dyes and forces into intensity due to mechanical deformation. By exploiting the optical properties of the constitutive materials and using machine learning, we infer spatiotemporal, haptic information from light that is read by an image sensor. We demonstrate the capabilities of our system in various assemblies to estimate temperature, contact location, normal and shear force, gestures, and damage from temporal snapshots of light coming from the entire haptic sensor with errors <5%.