GeXSe (Generative Explanatory Sensor System): A Deep Generative Method for Human Activity Recognition of Smart Spaces IOT

Abstract

Effective sensemaking is crucial across various domains, where trust in system outputs is essential, highlighting a significant challenge in relying solely on sensor-based activity recognition. This limitation underscores the need for interpretable models that process raw data and provide comprehensible insights. Our work introduces generative explanatory sensor system (GeXSe), a novel multitask framework that jointly models raw sensors for classification while also generating grounded visual explanations. At the core of GeXSe lies a parallel multibranch multilayer perceptron fast Fourier convolution (PMB-MLP-FFC) module tailored for multimodal sensor fusion and explanation generation. PMB-MLP-FFC extracts interpretable features optimized for both tasks through multibranch parallel convolutions and Fourier transforms. We validated GeXSe across three diverse public datasets of daily activities recorded by camera, microphone, motion, and environmental sensors. Results show superior activity recognition over baseline models. Furthermore, human evaluation studies confirm the generated visual explanations enhance understanding and trust compared to purely sensor-based outputs.