1989–1995

Multimodal Affective Interfaces and Measurement

Researchers established multimodal emotion recognition by combining facial expressions and physiological cues within human computer interaction. Studies utilized standardized stimuli and scalable annotation methods to support early emotion aware computing systems. Foundational mood as information frameworks reframed emotion as essential context for decision making. Compact affect measures and normative image sets provided reproducible resources that anchored later developments.

Period 1989-1995: Multimodal emotion recognition and psychometric measurement unified sensing, interpretation, and interaction design[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19]. Foundational mood framing, compact scales, and standardized image sets established reproducible resources that shaped later affective computing[20][14][1][21][22].

1996–2013

Multimodal Sensing and Emotion Recognition

Research coalesced around multimodal sensing for emotion-aware human-computer interaction to achieve real-time interpretation and adaptive responses. Visual feature representations for facial expression recognition relied on texture-based and transform-based descriptors like Local Binary Patterns and Gabor wavelets. Standardized datasets and benchmarks underpinned evaluation, while affective semantics and grounding supported cross-domain learning. Furthermore, automatic extraction of emotion trajectories from video and physiological signals enabled time-sensitive adaptations in intelligent systems.

Period 1996-2013: Multimodal systems integrated facial, vocal, and physiological cues using texture descriptors, datasets, and semantic frameworks to enable real-time emotion recognition[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17]. Landmark studies and standardized resources like CK+, IAPS, and IEMOCAP established foundational methodologies and benchmarks that continue to drive multimodal affective computing[18][14][12][19].

2014–2023

Multimodal and Self-Supervised Affective Computing

Researchers transitioned toward affective understanding in unconstrained environments by leveraging multimodal fusion of facial, vocal, physiological, and contextual cues. Deep representation learning, end-to-end architectures, and standardized benchmarks drove robust emotion decoding across diverse real-world applications. Subsequent developments integrated self-supervised multi-task objectives, graph neural networks, and attention mechanisms to enhance signal robustness and cross-subject generalization. These advancements established unified frameworks capable of processing complex affective signals with improved accuracy.

Period 2014-2020: Deep multimodal fusion and EEG decoding merged visual, vocal, and neurophysiological cues for unconstrained affective computing[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15]. AffectNet, OpenFace 2.0, EEGNet, and GeMAPS standardized benchmarks and feature extraction to shape modern affective research[16][17][18][19]. Period 2021-2023: End-to-end architectures fuse facial, EEG, and speech cues via graph networks, attention, and domain adaptation to improve emotion recognition[20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36]. Landmark datasets, self-supervised objectives, regularized graph models, and attention networks shape robust multimodal affective computing[20][37][23][38].

Multimodal Affective Interfaces and Measurement era

Peter J. Lang and 3 others

Peter J. Lang[1] conducted research while affiliated with University of Florida[5][6]. Through the publications Looking at pictures: Affective, facial, visceral, and behavioral reactions[10] and International Affective Picture System (IAPS) : Technical Manual and Affective Ratings[11], this work provided compact affect measures and normative image sets that anchored reproducible resources for emotion-aware systems. Margaret M. Bradley[2] performed investigations at University of Florida[5][6]. The study Looking at pictures: Affective, facial, visceral, and behavioral reactions[10] contributed to multimodal emotion recognition by combining facial expressions and physiological cues within human-computer interaction. Alfons O. Hamm[3] pursued studies while at Justus-Liebig-Universität Gießen[7][8]. The paper Looking at pictures: Affective, facial, visceral, and behavioral reactions[10] helped establish standardized stimuli and scalable annotation methods to support early emotion-aware computing systems. Mark K. Greenwald[4] worked across University of Florida[5] and Johns Hopkins University[9]. The research presented in Looking at pictures: Affective, facial, visceral, and behavioral reactions[10] advanced mood as information frameworks by reframing emotion as essential context for decision-making.

Multimodal Sensing and Emotion Recognition era

Pj. Lang and 2 others

Pj. Lang[12] advanced the era by providing standardized datasets and benchmarks that underpinned evaluation for affective computing systems. Through International affective picture system (IAPS) : affective ratings of pictures and instruction manual[18], this work established affective semantics and grounding to support cross-domain learning in emotion-aware human-computer interaction. Matti Pietikäinen[13], working at University of Oulu[15] and Oulu University of Applied Sciences[16], contributed foundational visual feature representations for facial expression recognition. Through Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions[19], this research utilized texture-based descriptors like Local Binary Patterns to achieve real-time interpretation of dynamic textures. Guoying Zhao[14], working at University of Oulu[15] and Chinese Academy of Sciences[17], further expanded visual feature representations for facial expression recognition. Through Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions[19], this work applied Local Binary Patterns to dynamic texture recognition and facial expressions, advancing multimodal sensing and emotion recognition.

Multimodal and Self-Supervised Affective Computing era

Bao‐Liang Lu and Wei‐Long Zheng

Bao‐Liang Lu[20] conducted research while affiliated with Shanghai Jiao Tong University[22] and the Shanghai Municipal Education Commission[23]. Through Emotional state classification from EEG data using machine learning approach[25] and Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks[24], this work advanced affective understanding in unconstrained environments by identifying critical neural frequency bands and applying deep representation learning to decode emotional states from EEG signals. Wei‐Long Zheng[21] also advanced the field while working at Shanghai Jiao Tong University[22] and the Shanghai Municipal Education Commission[23]. Their contribution was demonstrated in Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks[24], which established robust emotion decoding capabilities through advanced deep neural network architectures applied to complex physiological signals.