Language grounding in Conversational Agents

Lead Research Organisation: Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences


Text-only as well as multimodal conversational agents have witnessed an unprecedented level of interest among the AI community in recent years especially with the emergence of deep learning techniques in natural language processing and computer vision. Despite the ground-breaking work in the field, most current deep learning models rely on co-occurrence of tokens in large text corpora and generate text void of meaning. This work aims to develop conversational agents capable of comprehending the real-world meaning of tokens with respect to knowledge-context and visual-context of multimodal conversations and generate responses accordingly. We especially focus on a smart fusion between graph-structured (such as knowledge-graphs, scene-graphs) representations and language models.

Planned Impact

The Centre will have immediate short-term impacts on people skills and innovation pipeline, alongside key advances in scientific knowledge and techniques in Robotics and Autonomous Systems (RAS). With the strength of the programme's training emphasis on safety and responsible research and innovation (RI), we also target longer term economic and societal benefits.

Economy: It is estimated that the application of advanced robotics could generate a potential worldwide economic impact of $1.7-4.5 trillion by 2025 per year by 2025 (McKinsey). Over the last 5 years, the UK has witnessed significant new investments in robotics from both Government with the £4.7B Industrial Strategy Challenge Fund (ISCF), spawning parallel investments from industry. An example is the EPSRC £18M investment in the EPSRC ORCA Hub in RAS, led by Heriot-Watt, leveraging a further £18.5M industry investment. The potential economic impact of RAS, however, is hampered by a massive skills shortage, which the proposed CDT-RAS would address. The new jobs created by these investments will require highly specialised, yet interdisciplinary and industrially relevant skill-sets. The CDT-RAS is well positioned to supply the UK workforce in this growing area, through strong links with industry through its extensive CDT-RAS Project Partners network and through a training emphasis on 'innovation-ready' graduates. For example, our CDT-RAS students will have the opportunity to grow into industrial leaders of tomorrow through direct experience and company placements, as well as, through the CDT's extensive support for commercialisation and start-ups.

Society: Robotic and autonomous systems have already been identified by the UK government as a key component to enable safer working conditions for 'dull, dirty and dangerous' tasks in extreme and challenging environments such as offshore, nuclear, mining, and space. Moreover, there are disruptive opportunities for RAS to contribute to cost-effective and safer construction, transport, and manufacturing and improved quality-of-life through healthcare and assisted living. CDT-RAS training focuses on interdisciplinary, cross-cutting, yet responsible research and innovation to allow our future leaders to develop techniques and technologies that will have impacts in new areas, beneficially improving society beyond what we can already imagine. We will develop autonomous systems and AI enablers that are transparent to developers and end-users alike. This will allow robots and machines to work seamlessly in society both individually and in teams and comply with regulations, such as the EU General Data Protection Regulations (GDPR) and emerging IEEE standards, such as P7001 for Transparency for Autonomous Systems (for which Centre academics are members of the working group).

Science: CDT-RAS students will benefit from i) a critical mass of over 50 experienced supervisors, ii) the brand new facilities of the £27.5M National ROBOTARIUM and earlier £8M EPSRC equipment investments, as well as, iii) opportunities for international scientific and industrial lab placements. The Centre will realise scientific advancement and impact, crucial to enabling safe interaction between RAS, humans and the environment, including soft robotics, bio-inspired systems, human-robot interaction, swarms and collaborative robotics including human-robot teaming, sensing, embedded control, multi-agent decision making and maritime field robotics. The impact of the resultant research will be strengthened through top-venue publications and conference presentations, utilising student presentation/writing skills honed during the CDT-RAS training. Impact will also come through outreach such as international student robot competitions, public engagement activities such as science festivals and CDT-RAS hosted international researcher visitors and workshops.


10 25 50

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/S023208/1 01/10/2019 31/03/2028
2591081 Studentship EP/S023208/1 01/10/2020 30/09/2024 Supun Hemanthage