Computational modelling of culture-specific social facial signals with transference to digital agents

Strategic priority area-Psychology
Keywords-social perception, psychophysics, apology
Facial expressions are a powerful tool for social communication because they can transmit myriad information, including emotions (e.g., Jack et al., 2016) and personality traits (e.g., Gill et al., 2014). A longstanding goal is to understand the system of facial communication-i.e., what specific facial movements transmit what messages to whom and in what culture (e.g., Darwin, 1872/1999). One influential theory posits that facial expressions comprise core cross-cultural signals plus culture-specific accents that hinder cross-cultural communication while conferring in-group advantages (e.g., Elfenbein & Ambady, 2002). Relatedly, another theory (e.g., Tsai, 2007) posits that facial expressions of culture-specific ideal affect (e.g., USA vs. China-broad vs. content smiles) influence key social judgments (e.g., intelligence, leadership; Park et al., 2020; Krys et al., 2014) that could compound cross-cultural miscommunication. Yet, understanding the complex interplay of dynamic facial signals and their impact on social perception remains empirically challenging because the face is highly complex (e.g., Jack & Schyns, 2017). We will address this empirical challenge using our state-of-the-art computer graphics-based 3D face generator, rigorous data-driven methods, and precise information-theoretic analyses tools (e.g., Jack et al., 2012; Liu et al., 2021; Zhan et al., 2021) to mathematically model the dynamic facial cues (movements/shape/complexion) that drive key social judgments (emotion and social traits) in individuals in distinct cultures (East Asian, Western).

Project impact
Science. This project will provide a new, culturally nuanced account of social facial signalling with direct impact in central debates/theories, including the cultural universality/variability, ontology, and function of facial expressions (e.g., Elfenbein & Ambady, 2002; Jack et al., 2016; Shariff & Tracy, 2011) and their intersection with face identity cues (shape/complexion) in shaping social perception (e.g., Gill et al., 2014).
Society. This work is directly relevant to several facets of society, particularly with increasing globalisation, cultural integration, and the digital economy. Understanding cross-cultural similarities and differences in facial expressions can directly impact everyday cross-cultural communications to avoid/reduce misunderstandings and inform new remote/virtual technologies. Therefore, the project will be of immediate interest to the public with directly applicable results. Further, digital agents are now embedded into human society including for companionship, health care, education, and entertainment, where social skills are vital (Morency, 2015). By transferring the culturally and socially nuanced facial signals to digital agents, this project aims to enhance their social communication capabilities, utility, accessibility, and global marketability.

Training plan
Year 1. In-house training in specialist skills (e.g., advanced programming: Matlab/Python; 3D/4D face signal modelling; specialist toolboxes), literature reviews to develop knowledge, submit ethics applications, collect pilot data. The student will be expected to provide direct inputs in shaping the direction of the project using their acquired knowledge of the literature, and to participate in 1-2-1 and group laboratory meetings to discuss and review these research directions.
Year 2. Continue data collection, build computational models of facial signals across cultures, conduct primary analysis, prepare early work for submissions to international conferences to obtain feedback.
Year 3-3.5. Finalise data collection and analysis, prepare submission of manuscripts to peer- reviewed broad-audience journals; transfer facial signals to digital agents, evaluate impact, and prepare additional submission of manuscripts to peer-reviewed social robotics journals/conferences.