A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder

TLDR

Detecting anomalous executions is crucial for reducing hazards in assistive manipulation, yet fusing high‑dimensional, heterogeneous multimodal signals remains a challenging problem. The study proposes an LSTM‑based variational autoencoder that fuses multimodal signals and reconstructs their expected distribution using a progress‑based varying prior. Anomaly detection is performed by computing a reconstruction‑based anomaly score from the LSTM‑VAE and flagging executions when this score exceeds a state‑based threshold. On 1555 feeding executions with 12 anomaly types, the detector achieved an AUC of 0.8710, outperforming five baseline methods, and demonstrated effective anomaly detection across 17 raw sensory signals without extensive feature engineering.

Abstract

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem for model-based anomaly detection. We introduce a long short-term memory-based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution by introducing a progress-based varying prior. Our LSTM-VAE-based detector reports an anomaly when a reconstruction-based anomaly score is higher than a state-based threshold. For evaluations with 1555 robot-assisted feeding executions, including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve of 0.8710 than 5 other baseline detectors from the literature. We also show the variational autoencoding and state-based thresholding are effective in detecting anomalies from 17 raw sensory signals without significant feature engineering effort.