Predicting Affect Dynamics during Naturalistic Viewing from Sentiment Analysis and Brain Functional Connectivity

Affective experiences are critical components of human behavior, guiding our thoughts, feelings, and actions. Prior research has largely focused on dimensional theories of emotions, such as valence and arousal, which rely on self-reported sentiments and behavioral tasks using unimodal stimuli like pictures and faces. However, obtaining continuous measures of affective experience in naturalistic environments presents significant challenges due to the complexity and dynamics of the real-world context. In this study, we explore the feasibility of predicting individuals' dynamic affective experience during naturalistic viewing using semantic information extracted from annotated descriptions of the movie, as well as brain functional connectivity. By applying fine-grained sentiment analysis, we derive a continuous representation of the movie's sentiment during the movie-viewing experience. We then correlate these sentiments with behavioral valence and arousal ratings from an independent group of 60 participants and previously collected fMRI data from 17 participants who watched a TV show episode in the scanner (Chen et al., 2017). Our results suggest that automated sentiment analysis can effectively predict subjective valence on an event-level basis, even though it may not be a reliable predictor of affective experience at the sentence level. Meanwhile, we developed a more flexible approach to clustering sentiment scores based on meaningful shifts in the narrative such as topic, time, location, which provides a more accurate and contextualized analysis of sentiment. These findings provide new insights into how individuals' dynamic feelings of valence during movie-watching can be measured and predicted, contributing to the growing body of literature and methods in natural language processing, affective computing, emotion processing, and whole-brain functional connectivity.