Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection

TLDR

Daily activities in continuous streams have variable durations, causing random transitions that preclude fixed‑duration snapshots. The study proposes a multilayer bi‑LSTM framework for multimodal sensor fusion to detect daily activities and falls. Using continuous data from FMCW radar and wrist, waist, ankle inertial sensors, the bi‑LSTM performs soft feature fusion and two hard‑fusion methods based on confusion matrices, and a hybrid scheme combining both, evaluated with a leave‑one‑participant‑out protocol. The hybrid‑fusion bi‑LSTM achieves about 96 % accuracy, reduces accuracy variance by 18.1 % and raises the worst‑case accuracy by 16.2 % across participants.

Abstract

This paper presents a framework based on multilayer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities' patterns and high-risk events such as falls. The data collected in this work are continuous activity streams from FMCW radar and three wearable inertial sensors on the wrist, waist, and ankle. Each activity has a variable duration in the data stream so that the transitions between activities can happen at random times within the stream, without resorting to conventional fixed-duration snapshots. The proposed bi-LSTM implements soft feature fusion between wearable sensors and radar data, as well as two robust hard-fusion methods using the confusion matrices of both sensors. A novel hybrid fusion scheme is then proposed to combine soft and hard fusion to push the classification performances to approximately 96% accuracy in identifying continuous activities and fall events. These fusion schemes implemented with the proposed bi-LSTM network are compared with conventional sliding window approach, and all are validated with realistic "leaving one participant out" (L1PO) method (i.e. testing subjects unknown to the classifier). The developed hybrid-fusion approach is capable of stabilizing the classification performance among different participants in terms of reducing accuracy variance of up to 18.1% and increasing minimum, worst-case accuracy up to 16.2%.