EST-TSANet: Video-Based Remote Heart Rate Measurement Using Temporal Shift Attention Network and ESTmap

Abstract

Remote photoplethysmography (rPPG)-based heart rate (HR) measurement approaches have attracted increasing attention recently. In recent deep-learning-based approaches, the inherent redundancy and noise in video frame contents are urgent issues to be addressed. Some studies utilize spatial–temporal maps (STmaps) to encode the video clips into efficient spatial–temporal representations. Since the subtle physiological signals are vulnerable to external factors such as head movement and unstable illumination, how to construct a better physiological representation is a significant factor affecting the method’s performance. In addition, existing STmap-based methods always neglect temporal modeling between video clips, which is very important for stable video HR estimation. To address the above problems, in this article, we propose an effective temporal shift attention network with enhanced STmap as its input (EST-TSANet). We first propose an enhanced STmap (ESTmap), which combines both local and global physiological information to construct a better physiological representation. To establish the temporal relationship between multiple STmaps, we construct a split temporal shift block (STSB) and leverage temporal shift to perform efficient temporal modeling. Besides, considering the characteristics of STmap, a novel plug-and-play module named dimension-specific attention module (DSAM) is designed to optimize the information learning specifically for each dimension. Extensive experiments on the public VIPL-HR and UBFC-rPPG datasets show that our proposed EST-TSANet outperforms the state-of-the-art (SOTA) methods.