IEEE Transactions on Affective Computing · 2022 · 12 citations · 37 references
ComputingComputer ScienceComputer VisionNatural Language ProcessingVisual AttentionAttention NetworksDeep LearningElectroencephalogram AnalysisElectroencephalographyEmotion RecognitionText SummarizationAttentionDeep Reinforcement LearningModality AlignmentMultimodal SummarizationNeural SummarizationSummarization DatasetsTemporal AlignmentVideo Summarization
Video summarization is the process of selecting a subset of informative keyframes to expedite storytelling with limited loss of information. In this article, we propose an EEG-Video Emotion-based Summarization (EVES) model based on a multimodal deep reinforcement learning (DRL) architecture that leverages neural signals to learn visual interestingness to produce quantitatively and qualitatively better video summaries. As such, EVES does not learn from the expensive human annotations but the multimodal signals. Furthermore, to ensure the temporal alignment and minimize the modality gap between the visual and EEG modalities, we introduce a Time Synchronization Module (TSM) that uses an attention mechanism to transform the EEG representations onto the visual representation space. We evaluate the performance of EVES on the TVSum and SumMe datasets. Based on the rank order statistics benchmarks, the experimental results show that EVES outperforms the unsupervised models and narrows the performance gap with supervised models. Furthermore, the human evaluation scores show that EVES receives a higher rating than the state-of-the-art DRL model DR-DSN by 11.4% on the coherency of the content and 7.4% on the emotion-evoking content. Thus, our work demonstrates the potential of EVES in selecting interesting content that is both coherent and emotion-evoking.
37
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Reinforcement Learning: An Introduction
Jeffrey D. Johnson, Jinghong Li, Zengshi Chen · Neurocomputing · 2000 · 8.7K citations
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Sander Koelstra, Christian Mühl, Mohammad Soleymani et al. · IEEE Transactions on Affective Computing · 2011 · 4.8K citations · Full text
Nursing, Neuroscience, Psychology +17