Understanding emotion through multimedia

Abstract

We conducted an experiment to determine the abilities of hearing-impaired and normal-hearing people to recognize intended emotions conveyed in four types of stimuli: a drum performance, a drum performance accompanied by a drawing expressing the same intended emotion, and a drum performance accompanied by one of two types of motion pictures. The recognition rate was the highest for a drum performance accompanied by a drawing even though participants in both groups found it difficult to identify the intended emotion because they felt the two stimuli sometimes conveyed different emotions. Visual stimuli were especially effective for performances whose intended emotions were not clear by themselves. The difference in ability to recognize intended emotions between the hearing-impaired and normal-hearing participants was insignificant. The results of this and a series of experiments will enable us to better understand the similarities and differences between how people with different hearing abilities encode and decode emotions in and from sound and visual media. We should then be able to develop a system that will enable hearing-impaired and normal-hearing people to play music together.