Expressive behaviour as a social signal for socially-competent human-robot interaction: Expre-ss
Lead Research Organisation:
Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences
Abstract
The travel grant would support work creating mechanisms for embodied social agents that apply theory of mind (ToM) capabilities to the detection and management by the agent of its own modelled affective state and the expressive behaviour it generates as a result. This will support the development of more socially competent agents suitable for a wider range of applications. It flows from work currently being conducted in the project SoCoRo (EP/N034546/1), investigating the use of an expressive robot to help high-functioning adults with autism with social signal processing.
Social agents are graphically or robotically embodied entities designed to act competently and successfully in everyday human social environments. These could be homes, schools, workplaces, or more public spaces, like shopping malls and museums. It is well-established that humans impute social agency to such embodied entities and therefore assume competence in the social signals humans use themselves to regulate interaction and to manifest what they feel or think. For this reason, researchers in embodied social agents have long worked on agent expressive behaviour - facial expressions, vocalisations, gestures, posture, both for graphical and robotic agents.
Generating expressive behaviour (as distinct from scripting via pre-annotated dialogue items) requires an architecture that can model affective responses. Such an architecture should be based on sound psychological theory, but should move beyond the rather simple idea that the social agent merely expresses its own internal state. A smile can mean very many things other than an internal state of happiness and an employee being dressed down by their boss is unlikely to express the anger they may feel. Social signals are social because they take into account the social context and the impact such signals may make on interaction partners. Social agents that motivate in education, offer support to the elderly, are involved in role-play, in training applications or in therapy all need this capability.The travel grant will support an investigation of generic mechanisms that can be implemented in existing social agent architectures.
The applicant will work with leading theorists at University of Geneva and then with the builders of the two leading agent architectures in University of Lisbon and University of Southern California.
Social agents are graphically or robotically embodied entities designed to act competently and successfully in everyday human social environments. These could be homes, schools, workplaces, or more public spaces, like shopping malls and museums. It is well-established that humans impute social agency to such embodied entities and therefore assume competence in the social signals humans use themselves to regulate interaction and to manifest what they feel or think. For this reason, researchers in embodied social agents have long worked on agent expressive behaviour - facial expressions, vocalisations, gestures, posture, both for graphical and robotic agents.
Generating expressive behaviour (as distinct from scripting via pre-annotated dialogue items) requires an architecture that can model affective responses. Such an architecture should be based on sound psychological theory, but should move beyond the rather simple idea that the social agent merely expresses its own internal state. A smile can mean very many things other than an internal state of happiness and an employee being dressed down by their boss is unlikely to express the anger they may feel. Social signals are social because they take into account the social context and the impact such signals may make on interaction partners. Social agents that motivate in education, offer support to the elderly, are involved in role-play, in training applications or in therapy all need this capability.The travel grant will support an investigation of generic mechanisms that can be implemented in existing social agent architectures.
The applicant will work with leading theorists at University of Geneva and then with the builders of the two leading agent architectures in University of Lisbon and University of Southern California.
Planned Impact
Organisations applying embodied social agents in health, domestic support, training, education and therapy will benefit from the availability of more socially-competent embodied agents since these should both function more effectively, especially during long-term interaction, and be more acceptable to the humans with whom they interact.
This should support an increase in deployment into socially sensitive application domains with positive social impact on those interacting with such agents.
Companies manufacturing social robots will benefit from new mechanisms to generate appropriate expressive behaviour in their products.
Colleagues working in public engagement in robotics and dealing with some of the panic reactions generated by current hype will benefit from an example of socially useful robotics research.
Students of the HWU and University of Glasgow DTC will benefit from research-related teaching materials in their programmes.
This should support an increase in deployment into socially sensitive application domains with positive social impact on those interacting with such agents.
Companies manufacturing social robots will benefit from new mechanisms to generate appropriate expressive behaviour in their products.
Colleagues working in public engagement in robotics and dealing with some of the panic reactions generated by current hype will benefit from an example of socially useful robotics research.
Students of the HWU and University of Glasgow DTC will benefit from research-related teaching materials in their programmes.
Organisations
People |
ORCID iD |
Ruth Aylett (Principal Investigator) |
Publications
Aylett Ruth
(2021)
Living with Robots: What Every Anxious Human Needs to Know
Iwaya LH
(2023)
On the privacy of mental health apps: An empirical investigation and its implications for app development.
in Empirical software engineering
McKenna P
(2020)
"Sorry to Disturb You"
Description | A new approach to translating a modelled emotional state fo a graphical agent into graphically naturalistic behaviour - for example facial colour changes, sweating |
Exploitation Route | The University of Geneva Virtual Human is a good test bed for other research groups |
Sectors | Creative Economy,Healthcare,Culture, Heritage, Museums and Collections |
Description | Article in BBC Science Focus Magazine |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Invited article in BBC Science Focus Why Robots will not take over any time soon.. |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.sciencefocus.com/future-technology/future-robots-society/ |
Description | Interview on BBC World Service programme Tech Tent |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Short interview on BBC World Service programme Tech Tent on robots and human-likeness |
Year(s) Of Engagement Activity | 2022 |
Description | Interview with BBC Midlands |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Public/other audiences |
Results and Impact | 10 min interview on the BBC Midlands radio station |
Year(s) Of Engagement Activity | 2021 |
Description | Invited talk, ACM International Conference on Intelligent Virtual Agents 2022 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Invited talk to a conference of 100+ people in Portugal |
Year(s) Of Engagement Activity | 2022 |
URL | https://ivaconference2022.ualg.pt/program/invited-speakers/ |
Description | Online interview on Owen Jones show 'Will robots take over the world?' |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | 30 mins interview on robotics on the online Owen Jones show |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.youtube.com/watch?v=Gs2k4CSbq_g |
Description | Online seminar for University of Geneva |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | 50 researchers attended an online seminar Apr 1st at University of Geneva: title From facial expressions to social signals for useful embodied social agents |
Year(s) Of Engagement Activity | 2020 |
URL | https://www.campusbiotech.ch/en/node/676 |
Description | Panel/talk : Art of Possible - The future of emerging tech in business - opportunities vs threats |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | An online panel and keynotes organised by Technology Scotland to update industry about the impact of AI technologies |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.youtube.com/watch?v=jqNoRv-Ya9c |
Description | Podcast fro New Book network |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Podcast with my colleague Patricia Vargas on the New Book Network |
Year(s) Of Engagement Activity | 2021 |
URL | https://newbooksnetwork.com/living-with-robots |
Description | REDDIT Robotics forum |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Invited Keynote in the REDDIT Robotics Forum on social robotics |
Year(s) Of Engagement Activity | 2023 |
URL | https://www.reddit.com/r/robotics/comments/tyz5ho/announcing_keynote_speakers_for_the_reddit/ |
Description | Seminar for Edinburgh Napier |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | Invited seminar to Edinburgh Napier Robotics research group 19 Jan 2022 |
Year(s) Of Engagement Activity | 2022 |
Description | Virtual seminar in Talking Robotics series |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Seminar presentation in the Series 'Talking Robotics' Nov 5th 2021 |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.youtube.com/watch?v=zOXKr5-ik4w |