From data and theory to computational models of more effective virtual human gestures

Metaphors play a central role in how people think about and convey abstract ideas. In fact, 'conveying an idea' is a form of metaphor, as if an idea is a physical object that can be physically transferred. We speak of getting ideas across to someone, of giving someone an idea, of possessing ideas (having) or grasping an idea. Research has extensively documented a wide array of metaphor use in our language and more fundamentally in our thought processes

Such metaphors are also reflected in our gesturing. Specifically, metaphoric gestures provide a physical manifestation for these metaphors. Speakers mimic holding an imaginary object to suggest an idea. The importance of the idea may be indicated by the size of this imaginary object. Speakers also use the physical space to structure their discourse. For example, contrasting differences between ideas, this versus that, may be conveyed by gestures that refer to the ideas as physical objects located in distinct locations, thus differences in location imply differences in the ideas. Particular locations in space can also derive meaning from the underlying metaphor, For example, a common metaphor for time is that events in time are points in space along a line so that, in some cultures, events located on the left are understood as being in the past while events on the right are in the future.

Gestures are known to influence understanding and information recall: listeners are more likely to learn and remember an instruction when it is accompanied by gesture (Cook & Goldin-Meadow, 2006). They also impact interpersonal attitudes of listeners towards speakers, by influencing inferences about personal and social features (Goldin-Meadow & Alibali, 2013). In particular, speakers who use metaphoric gestures are judged more persuasive and competent than those who do not (Maricchiolo et al. 2009). By associating visual features to abstract concepts and laying them out in space, metaphors and metaphoric gestures enable the use of spatial reasoning mechanisms on abstract concepts and facilitate comprehension (Beaudoin- Ryan & Goldin-Meadow, 2014; Calbris, 2011; Kendon, 2004; McNeill, 2005).

The focus of this research is the computational modeling of the relation between metaphor and gesture. Beyond fostering a deeper understanding of the relation between abstract concepts, metaphor and physical gesture, the main goal of this work is to improve Human-Computer Interaction (HCI). HCI research has explored how we can make the machine more human-like. Specifically, virtual humans, autonomous agents with anthropomorphic features and behaviors engage users in face-to-face interactions, using the same verbal and nonverbal behaviors as humans. Virtual human technology has proven beneficial in a wide range of applications, including virtual patients that teach doctors how to break bad news to patients, tutors that motivate students to learn, assistants to the elders and persuasive health interventions seeking behavioral change.

This proposal is a multidisciplinary effort to computationally model metaphoric gestures and systematically evaluate their effect in human-virtual human interaction. We also see this work as having a significant impact on the design of social robots that can interact with people.

The impact of this work in part stems from the importance of gestures in human social interaction and that human social interaction is increasingly between human and facsimiles of people. Graphics based virtual humans and social robots with anthropomorphic features and behaviors engage users using the same verbal and nonverbal behaviors that people use when interacting with each other. These characters are used to help patients adhere to healthy behavior, provide companions for the elderly, assess a user's mental states such as depression, teach job interview skills to adults with autism spectrum disorder, teach doctors better bed side manner and train cross-cultural negotiation skills in the military. Virtual characters are also being deployed in web and messaging technologies to engage users and enliven the interaction. These technologies exploit the fact that nonverbal behaviors of participants powerfully influences face-to-face interaction.

The key goal of the research is to fundamentally improve the nonverbal capabilities of these artificial social agents by providing the capacity for meaningful gesturing grounded in the pervasive role of metaphor in human thought, language and social interaction. This will fundamental enhance the effectiveness of these social agents in the wide range of applications that use these persuasive technologies. The proposed research will enhance knowledge of the relation between metaphors and gestures, develop models of this relation, improve technologies for realizing gestures in aritificial social agents and empirical evaluate these technologies in user studies.

Our impact strategy is twofold. First is to ensure the resulting models and technologies are promulgated, specifically to inform and facilitate the work of researchers and developers of artificial social agents. Second is to provide a deeper understanding of how metaphors relate to gestures and how the resulting gestures influence people. This understanding is both a basic research question as well as being necessary for effective application of the technology. It is the expected outcome of the proposed empirical work on a) how metaphors, including multiple metaphors, map into gestures and sequences of gestures and b) whether these gestures, when realized in an artificial agent, are understood by, and influence, people.

Impact tools
Publications: The results of this work will be published in technology conferences and journals such as Autonomous Agents and Multi-Agent Systems, Intelligent Virtual Agents and Human Robot Interaction. Because the data collected/analyzed, the resulting models and empirical studies will also be relevant to psychologists and linguists, we will publish these results in appropriate venues to reach psychologists and linguists, such as the journal Gesture.
Website: Here, the data collected, the models and software for driving gesturing with be available for download. It will be available separately as well as part of the virtual human toolkit, an open source toolkit for realizing virtual humans. On the website, visitors will be able to vividly experience a virtual human, who can process dialog input from users and illustrate the gesturing.
Training: This work will also be integrated into the University of Glasgow's UKRI Centre for Doctoral Training in Socially Intelligent Artificial Agents. Specifically the work will be used in classwork and research projects, educating the next generation of researchers in technology, helping to disseminate the technology in their papers and future employment.

Public Dissemination strategy
We will also increase the public's awareness through public engagement events and news media. We will reference our website in all articles and in press releases about our research.