Predicting Emotion From Color Present in Images and Video Excerpts by Machine Learning

Abstract

This work aims at predicting emotion based on the colors present in images and video excerpts using a machine-learning approach. The purpose of this paper is threefold: (a) to develop a machine-learning algorithm that classifies emotions based on the color present in an image, (b) to select the best-performing algorithm from the first phase and apply it to film excerpt emotion analysis based on colors, (c) to design an online survey to check the accuracy of the annotations of the collected film data. In the first, three approaches to color extraction are tested, namely clustering colors into a palette of predefined colors, assigning colors to the RYB (Red, Yellow, Blue) model, and extracting a histogram of colors present in an image. This is based on image datasets containing color and emotion annotations. Classification is conducted using several algorithms, both deep learning and baseline artificial intelligence algorithms. The obtained results, under different configurations of parameters and training sets, are then presented. In the second part, the best-performing algorithm from the first phase is applied to film excerpt emotion analysis based on colors. This is followed by the third part, which is an online survey created to check the accuracy of the algorithm’s annotations to the collected film data by the questionnaire respondents. Further, a discussion of the results achieved is presented. Conclusions contain a summary of the results and further direction for improving the performance of the created algorithm.