Intelligent Multimodal Human Behavior Recognition using Inertial and Video Sensors

Abstract

Tracking human actions, especially in the field of medicine, has encouraged interest in computer vision, specifically for detecting gait anomalies to aid in rehabilitating patients with irregular patterns. In this paper, the proposed system employs vision-based and inertial-based modalities that enable the recording of human actions and encourage human-machine collaboration. The inertial sensors were preprocessed using a Butterworth filter, as its magnitude response is often monotonic and maximally flat in the passband, providing smoothness. Multiple key features, such as Mel Frequency Cepstral Coefficients (MFCC) and auto-regression coefficients have been extracted. On the other hand, human detection has been performed for vision-based data, and main features such as angles and velocities have been calculated. The final results have been fused using multimodal survey fusion. The fused data was then forwarded for optimization using yeo-johnson power optimizer. Artificial Neural Networks (ANN) have been trained to perform classification. Evaluation using the confusion matrix on the Heriot-Watt University of Sao Paulo (HWU-USP) dataset proficient of identifying nine complex actions showed an accuracy rate of 83.33%.