Studying Social Interactions with Audiovisual Transformations

Human social interactions are packed with social signals such as smiles, frowns, or vocal intonations. However, despite the massive advances identifying the specific characteristics of such social signals, researchers have not tested their causal effect in the wild, during interactions. This is because researchers do not have the tools to parametrically control social signals in real-time. That is the aim of the current research program: To leverage the recent advancements in voice and face transformation
algorithms, to develop, test, and share, a new methodological platform giving researchers the ability to manipulate participants' vocal
and facial signals in real time during unscripted interactions.
To do this, I will develop audiovisual transformation algorithms, informed by human perception, allowing us to dynamically control the signals produced by a participant during a conversation (e.g.increase/decrease their smiles or vocal emotions). I will integrate these algorithms to my experimental platform DuckSoup, a videoconference tool (similar to Zoom), that allows us to record social interactions efficiently, cross-culturally and online. I will use these tools in two experiments, where I will covertly transform participants' facial and vocal attributes in several emotional directions. To analyse these experiments, I will (1) extract the participants' facial, vocal and semantic behavior from the interaction recordings with deep-learning techniques, and (2) use Information-theoretic frameworks to investigate emergent physiological
dynamics. The combination of these interdisciplinary techniques will give me the unprecedented ability to measure physiological reactions in the wild and without the invasive aspect of sensors. Moreover, this project will provide the scientific community with an open source platform that will open the door to a new era of experimental paradigms, and unblock current limitations in social cognition research