Facial emotion recognition using multi-modal information

Abstract

Facial emotion recognition will become vitally important in future multi-cultural visual communication systems, for emotion translation between cultures, which may be considered analogous to speech translation. However so far the recognition of facial emotions is mainly addressed by computer vision researchers, based on facial display. Also detection of vocal expressions of emotions can be found in research work done by acoustic researchers. Most of these research paradigms are devoted purely to visual or purely to auditory human emotion detection. However we found that it is very interesting to consider both these auditory and visual information together, for processing, since we hope this kind of multi-modal information processing will become a datum of information processing in future multimedia era. By several intensive subjective evaluation studies we found that human beings recognise anger, happiness, surprise and dislike by their visual appearance, compared to voice only detection. When the audio track of each emotion clip is dubbed with a different type of auditory emotional expression, still anger, happiness and surprise were video dominant. However the dislike emotion gave mixed responses to different speakers. In both studies we found that sadness and fear emotions were audio dominant. As a conclusion, we propose a method of facial emotion detection by using a hybrid approach, which uses multi-modal information for facial emotion recognition.