Video Affective Content Analysis by Exploring Domain Knowledge

Film grammar is often used to invoke certain emotional experiences from audiences through changing visual, speech, and musical elements of videos. Such film grammar, referred to as domain knowledge, is of great importance for video affective content analysis but has not been thoroughly examined in research. In this paper, we propose an improved method for emotion recognition and regression from videos through exploring domain knowledge. We first investigate the domain knowledge of visual, speech, and musical elements, and infer probabilistic dependencies between elements and emotions from the summarized film grammar. Then, we transfer the summarized dependencies between elements and emotions as constraints, and formulate video affective content analysis, including both emotion recognition and emotion regression from video content, as a constrained optimization problem. Experiments on the LIRIS-ACCEDE database, the FilmStim database, and the DEAP database demonstrate that the proposed video affective content analysis method can successfully leverage well-established film grammar to improve emotion recognition and regression from video content.