LISI - Learning to Imitate Nonverbal Communication Dynamics for Human-Robot Social Interaction
Lead Research Organisation:
King's College London
Department Name: Engineering
Abstract
We are approaching a future where robots will progressively become widespread in many aspects of our daily lives, including education, healthcare, work and personal use. All of these practical applications require that humans and robots work together in human environments, where social interaction is unavoidable. Along with verbal communication, successful social interaction is closely coupled with the interplay between nonverbal perception and action mechanisms, such as observation of one's gaze behaviour and following their attention, coordinating the form and function of hand-arm gestures. Humans perform social interaction in an instinctive and adaptive manner, with no effort. For robots to be successful in our social landscape, they should therefore engage in social interactions in a human-like manner, with increasing levels of autonomy.
Despite the exponential growth in the fields of human-robot interaction and social robotics, the capabilities of current social robots are still limited. First, most of the interaction contexts has been handled through tele-operation, whereby a human operator controls the robot remotely. However, this approach will be labour-intensive and impractical as the robots become more commonplace in our society. Second, designing interaction logic by manually programming each behaviour is exceptionally difficult, taking into account the complexity of the problem. Once fixed, it will be limited, not transferrable to unseen interaction contexts, and not robust to unpredicted inputs from the robot's environment (e.g., sensor noise).
Data-driven approaches are a promising path for addressing these shortcomings as modelling human-human interaction is the most natural guide to designing human-robot interaction interfaces that can be usable and understandable by everyone. This project aims (1) to develop novel methods for learning the principles of human-human interaction autonomously from data and learning to imitate these principles via robots using the techniques of computer vision and machine learning, and (2) to synergistically integrate these methods into the perception and control of real humanoid robots. This project will set the basis for the next generation of robots that will be able to learn how to act in concert with humans by watching human-human interaction videos.
Despite the exponential growth in the fields of human-robot interaction and social robotics, the capabilities of current social robots are still limited. First, most of the interaction contexts has been handled through tele-operation, whereby a human operator controls the robot remotely. However, this approach will be labour-intensive and impractical as the robots become more commonplace in our society. Second, designing interaction logic by manually programming each behaviour is exceptionally difficult, taking into account the complexity of the problem. Once fixed, it will be limited, not transferrable to unseen interaction contexts, and not robust to unpredicted inputs from the robot's environment (e.g., sensor noise).
Data-driven approaches are a promising path for addressing these shortcomings as modelling human-human interaction is the most natural guide to designing human-robot interaction interfaces that can be usable and understandable by everyone. This project aims (1) to develop novel methods for learning the principles of human-human interaction autonomously from data and learning to imitate these principles via robots using the techniques of computer vision and machine learning, and (2) to synergistically integrate these methods into the perception and control of real humanoid robots. This project will set the basis for the next generation of robots that will be able to learn how to act in concert with humans by watching human-human interaction videos.
People |
ORCID iD |
Oya Celiktutan Dikici (Principal Investigator) |
Publications
Cetin E.
(2021)
Learning Routines for Effective Off-Policy Reinforcement Learning
in Proceedings of Machine Learning Research
Chithrra Raghuram V
(2022)
Personalized Productive Engagement Recognition in Robot-Mediated Collaborative Learning
Jiang J
(2023)
Generalised Bias Mitigation for Personality Computing
Salam H
(2024)
Automatic Context-Aware Inference of Engagement in HMI: A Survey
in IEEE Transactions on Affective Computing
Tan Viet Tuyen Nguyen
(2023)
The Impact of Robot's Body Language on Customer Experience: An Analysis in a Cafe Setting
Tan Viet Tuyen Nguyen
(2023)
A Multimodal Dataset for Robot Learning to Imitate Social Human-Human Interaction
Description | This project has shown that taking into account the nonverbal behaviours of the interaction partner, their speech and body movements, plays an important role in generating the target partner's co-speech gestures during dyadic interactions, and has resulted in novel algorithms and a dataset. The project outcomes will allow researchers and industry practitioners to generate behaviours for embodied agents (e.g., virtual agents, robots) beyond default behaviours and make them more socially aware and user attentive, which is important to ensure their acceptance and deployment in real-world applications. This project has implied that the generation of nonverbal gestures is interaction context dependent, e.g., education, healthcare, etc. The research team has investigated two different scenarios, namely, agreement and disagreement, which may occur frequently in our daily lives. The results have shown that using a single model over different scenarios does not necessarily yield the most reliable results. This opens new research questions about how to describe the interaction context, and how to incorporate it in nonverbal gesture generation. |
Exploitation Route | The knowledge, dataset, and algorithms generated during this project will be beneficial for the researchers working in human-human interaction and human-robot interaction. Specifically, the dataset collected will enable the development of novel methods for the analysis and synthesis of human behaviour as well as the generation of behaviours for embodied agents (not only robots but also virtual agents). |
Sectors | Digital/Communication/Information Technologies (including Software) |
Description | The models developed during the LISI project are currently being explored by a company to generate realistic nonverbal behaviours for virtual agents. |
Sector | Digital/Communication/Information Technologies (including Software) |
Description | Continual Learning Towards Open World Human-Robot Interaction |
Amount | £19,578 (GBP) |
Organisation | The Royal Society |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 02/2022 |
End | 02/2023 |
Description | Socially Acceptable Extended Reality Models and Systems |
Amount | £553,808 (GBP) |
Organisation | Innovate UK |
Sector | Public |
Country | United Kingdom |
Start | 09/2022 |
End | 09/2025 |
Title | LISI-HHI |
Description | The LISI-HHI (Learning to Imitate Social Human-Human Interaction) dataset comprises dyadic human interactions recorded in a wide range of communication scenarios. The dataset contains multiple modalities simultaneously captured by high-accuracy sensors, including motion capture, RGB-D cameras, eye trackers, and microphones. The LISI-HHI dataset is designed to be a benchmark for HRI and multimodal learning research for modelling intra- and interpersonal nonverbal signals in social interaction contexts and investigating how to transfer such models to social robots. |
Type Of Material | Database/Collection of data |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | N/A. |
URL | https://dl.acm.org/doi/10.1145/3568294.3580080 |
Title | Learning Routines for Effective Off-Policy Reinforcement Learning |
Description | This repository contains research code for the ICML 2021 paper Learning Routines for Effective Off-Policy Reinforcement Learning. We provide our implementations for the algorithms used to validate the effectiveness of the routine framework on the DeepMind Control Suite, namely: SAC, TD3, Routine SAC, Routine TD3, and FiGAR TD3. |
Type Of Material | Computer model/algorithm |
Year Produced | 2021 |
Provided To Others? | Yes |
Impact | No impact yet. |
URL | https://github.com/Aladoro/Routines4RL |
Description | A Computational Approach for Analysing Autistic Behaviour during Dyadic Interactions |
Organisation | King's College London |
Department | Institute of Psychiatry, Psychology & Neuroscience |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Developing the methodology, running experiments, and writing the paper. |
Collaborator Contribution | Providing the dataset, developing the methodology, and writing the paper. |
Impact | A peer-reviewed conference paper titled "A Computational Approach for Analysing Autistic Behaviour during Dyadic Interactions", which was published at the proceedings of the 12th International Workshop on Human Behaviour Understanding, organised as part of ICPR 2022. |
Start Year | 2021 |
Description | Learning Personalised Models for Automatic Self-Reported Personality Recognition |
Organisation | New York University Abu Dhabi |
Country | United Arab Emirates |
Sector | Academic/University |
PI Contribution | The PI collaborated with New York University, Abu Dhabi, to develop personalised machine learning models for modelling human behaviour, and their approach received the First Place Award in the Personality Recognition Track at ICCV Understanding Social Behaviour in Dyadic and Small Interactions Challenge 2021. The PI further collaborated with researchers from EPFL and New York University, Abu Dhabi, on a related topic. In particular, they focused on developing personalised models for productive engagement recognition during robot-mediated collaborative learning. |
Collaborator Contribution | The PI and the collaborators contributed equally to the development of the ideas and methods as well as the subsequent papers. |
Impact | So far, this collaboration resulted in 1) a peer-reviewed workshop paper published at the Proceedings of Machine Learning Research in 2022, Understanding Social Behavior in Dyadic and Small Interactions, 2) a peer-reviewed conference paper published at the proceedings of ACM International Conference on Multimodal Interaction in 2022, and 3) a peer reviewed workshop paper published at the adjunct proceedings of ACM International Conference on Multimedia 2023. |
Start Year | 2021 |
Description | Learning Personalised Models for Automatic Self-Reported Personality Recognition |
Organisation | Swiss Federal Institute of Technology in Lausanne (EPFL) |
Country | Switzerland |
Sector | Public |
PI Contribution | The PI collaborated with New York University, Abu Dhabi, to develop personalised machine learning models for modelling human behaviour, and their approach received the First Place Award in the Personality Recognition Track at ICCV Understanding Social Behaviour in Dyadic and Small Interactions Challenge 2021. The PI further collaborated with researchers from EPFL and New York University, Abu Dhabi, on a related topic. In particular, they focused on developing personalised models for productive engagement recognition during robot-mediated collaborative learning. |
Collaborator Contribution | The PI and the collaborators contributed equally to the development of the ideas and methods as well as the subsequent papers. |
Impact | So far, this collaboration resulted in 1) a peer-reviewed workshop paper published at the Proceedings of Machine Learning Research in 2022, Understanding Social Behavior in Dyadic and Small Interactions, 2) a peer-reviewed conference paper published at the proceedings of ACM International Conference on Multimodal Interaction in 2022, and 3) a peer reviewed workshop paper published at the adjunct proceedings of ACM International Conference on Multimedia 2023. |
Start Year | 2021 |
Description | IEEE VR 2023 Workshop on Multi-modal Affective and Social Behavior Analysis and Synthesis in Extended Reality (MASSXR) |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The PI contributed to the organisation of this workshop. The workshop showcased invited talks, technical papers, and featured a discussion panel. The key takeaways from the event closely mirrored the goals and outcomes of the project. |
Year(s) Of Engagement Activity | 2023 |
URL | https://sites.google.com/view/massxrworkshop2023 |
Description | Keynote talk |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The PI gave a talk titled "The Role of Human Behaviour in Building Social Robots" as part of the 2022 IEEE Ro-Man workshop on Machine Learning for HRI: Bridging the Gap between Action and Perception. After this talk, the PI was invited as a speaker to the annual summer school of the international project "Cross-modal Learning". |
Year(s) Of Engagement Activity | 2022 |
URL | https://ml-hri2022.ivai.onl |
Description | Nonverbal Communication Skills in Humans and Robots |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The PDRA and PI led the preparation of this special session proposal, which has been accepted to the IEEE International Conference on Robot & Human Interactive Communication 2022, where their project collaborators are part of the organisation committee. Strongly aligned with the goal of the LISI project, the key aim of this special session is to bring forth efforts to understand mechanisms underlying human-human nonverbal communication and introduce novel approaches to the design, development, and evaluation of robotic platforms inspired and driven by those mechanisms. The special session featured 5 technical papers, which were presented during the main conference as well as an industrial talk by Francesco Ferro, CEO and co-founder of PAL Robotics. Overall, the organisers received positive verbal feedback from the attendees. |
Year(s) Of Engagement Activity | 2022 |
URL | https://nonverbal-communication-skills.github.io |
Description | Socially-Informed AI for Healthcare: Understanding and Generating Multimodal Nonverbal Cues |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | This was a workshop organised by the PI and her collaborators, in conjunction with the ACM International Conference on Multimodal Interaction 2021. The key aim of this multidisciplinary workshop was to foster cross-pollination by bringing together computer scientists and social psychologists to discuss innovative ideas, challenges, and opportunities in the development of advanced algorithms relevant to the understanding, learning, and generation of multimodal nonverbal cues for creating the next generation of healthcare technologies. In total, there were four contributed papers presented in form of oral presentation, and the workshop also featured three keynote speakers, bringing multidisciplinary perspective and expertise from both academia and industry, which led to fruitful discussions between the participants afterwards. |
Year(s) Of Engagement Activity | 2021 |
URL | https://social-ai-for-healthcare.github.io |
Description | When Humans Turn the Tables on Machines |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Public/other audiences |
Results and Impact | The PI was part of a panel during the Festival of Disruptive Thinking, which was organised by King's College London. The event sparked interesting discussions between the panel members and participants, regarding the capabilities of AI and robots, creativity, ethics, and so on. |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.kcl.ac.uk/events/when-humans-turn-the-tables-on-machines |