Monitoring of Manufacturing through Natural Language

Lead Research Organisation: Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences

Abstract

The future of factories lies in distributed, mobile, industrial robots that are in-tune with the factory environment through sensors but also with the human operator's needs. This project will look at developing a Natural Language interface that allows the supervisor to 'Talk to their Factory', quickly getting updates, receiving alerts such as production delays and assigning roles for the mobile robots through fluid, natural interaction between human and machine. This will involve identifying the intent of the human through their natural language interaction and providing the right amount of information, whilst not overloading them unnecessarily. This project aligns with the EPSRC Industrial Strategy in terms of development of infrastructure and cuts across three grand challenges.
 
Description Throughout the past couple of years, we have focused on understanding natural language interaction between humans and robots, and the factors that influence interactions when humand and robot are co-situated in the same environment (e.g., working in close proximity of each other).
To further investigate this, we collected a dataset with hard-to-study language phenomena such as clarification questions (e.g., "pick that thing" - "Which one?") that are common in human-human collaboration but not so studied with robots.
More recently, we explored how the vision and language models that are widely-spread are not very good at handling visual tasks: they rely on the text to tell what is happening and often ignore the vision. This is critical for applications that use these models to handle both vision and language, particularly in robots (e.g., to decide what to do depending on what they see plus an instruction given by a human).
Exploitation Route The outcomes so far could be useful for applications in industry and manufacturing, although they could be applied to any scenario where the robot and the human need to collaborate closely together.
The use of vision and language models may be applicable to many different areas whenever an action needs to be decided from some visual and textual input.
Sectors Manufacturing, including Industrial Biotechology
 
Description Interview at the Robot Talk Podcast 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact I was interviewed by the Robot Talk Podcast for the RAS Network on human-robot collaboration. After my appearance, several people got in touch and my research has been more openly accessible.
Year(s) Of Engagement Activity 2021
URL https://ukrobotics.libsyn.com/episode-six-human-robot-collaboration-working-together