HUman-machine teaming for Maritime Environments (HUME)
Lead Research Organisation:
Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences
Abstract
The offshore energy and defence sectors share a vision of the future where people are taken out of harsh, extreme environments and replaced by teams of smart robots able to do the 'dirty and dangerous jobs', collaborating seamlessly as a team with each other and with the human operators and experts on-shore. In this new world, remote data collection, fusion and interpretation become central, together with the ability to generate transparent, safe actionable decisions from this data. We propose the HUME project (HUman-machine teaming for Maritime Environments), whose vision is to develop a coherent framework that enables humans and machines to work seamlessly as a team by establishing and maintaining a single shared view of the world and each other's intents through transparent interaction, robust to a highly dynamic and unpredictable maritime environments. The HUME project's ambitious and fundamental research programme will address fundamental research questions in the field of machine-machine and human-machine collaboration, robot perception and explainable autonomy and AI.
The Prosperity Partnership would build on a 20 year strategic relationship between SeeByte and HWU, with SeeByte originally a spin-out of Heriot-Watt University in 2001 and now a world-leader in maritime autonomy worldwide in the Oil & Gas and Defence sectors. This grant would facilitate a shift to lower TRL research and development, providing seeding for early-stage research that can have a broad, longer-term and more disruptive impact. This proposed work aims at establishing a durable model, through which SeeByte and HWU can remain connected to foster long-term research relationships on projects of interest, as they emerge in this rapidly changing field.
The Prosperity Partnership would build on a 20 year strategic relationship between SeeByte and HWU, with SeeByte originally a spin-out of Heriot-Watt University in 2001 and now a world-leader in maritime autonomy worldwide in the Oil & Gas and Defence sectors. This grant would facilitate a shift to lower TRL research and development, providing seeding for early-stage research that can have a broad, longer-term and more disruptive impact. This proposed work aims at establishing a durable model, through which SeeByte and HWU can remain connected to foster long-term research relationships on projects of interest, as they emerge in this rapidly changing field.
Publications

Chiyah Garcia, F. J.
(2021)
Self-Explainable Robots in Remote Environments

David A. Robb
(2023)
Explanation Styles for Trustworthy Autonomous Systems

Gavriilidis K
(2022)
Towards Explaining Autonomy with Verbalised Decision Tree States

Gavriilidis K
(2023)
A Surrogate Model Framework for Explainable Autonomous Behaviour

Gavriilidis, K.
(2021)
Plan Verbalisation for Robots Acting in Dynamic Environments

Katsakioris MM
(2021)
Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps

Konstantinos Gavriilidis
(2022)
Towards Explaining Autonomy with Verbalised Decision Tree States

Willners J
(2021)
From market-ready ROVs to low-cost AUVs

Winfield AFT
(2021)
IEEE P7001: A Proposed Standard on Transparency.
in Frontiers in robotics and AI
Description | The second year of the HUME project has seen significant progress made towards realising the program's vision of transforming high-risk work environments through the deployment of intelligent robotic teams. These robots are engineered to perform challenging tasks in hazardous conditions, working collaboratively with one another and with human operators who might be remotely located. The focus of this vision is the effective collection, integration, and analysis of remote data, which is essential for enabling the robots to make autonomous decisions. Key achievements this year include: (1) the development of sensor quality metrics for ensuring reliable and accurate data collection by the robots, (2) the integration of initial explainable planning capabilities with SeeByte's Autonomy and Mission Planner Neptune, (3) ongoing integration with the MIRIAM chatbot, enhancing the system's interactive capabilities, and (4) continued research and development in explainable autonomy, conducted through an aligned PhD project. |
Exploitation Route | The final year of the project will focus on identifying technology 'nuggets' that can be exploited by the project industry partner SeeByte, either to inform their future products or to integrate into the current product line. We have already identified online and collaborative mission planning as an area of interest for multi-vehicle planning. This is currently one of the main product lines of SeeByte and the current prototype is fully integrated in Neptune. |
Sectors | Aerospace Defence and Marine |
Description | The award has initiated a robotics training program for our industry partner's engineers. This initiative is designed to equip SeeByte's engineering team with the skills and knowledge required to effectively leverage the capabilities of the robotic systems developed on HUME. This not only enhances the operational expertise within SeeByte but also lays the groundwork for future exploitation and innovation. Findings from the award have been used to identify areas of interest for multi-vehicle planning, which has been integrated into one of the main product lines of SeeByte. Additional findings in the third year of the project will continue to inform their future products. |
First Year Of Impact | 2023 |
Sector | Aerospace, Defence and Marine |
Impact Types | Economic |
Description | Prosperity Partnership Fugro and Imperial College London |
Amount | £1,124,000 (GBP) |
Funding ID | UNITE: EP/X024806/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 04/2023 |
Description | Heriot-Watt University Prosperity Partnership with Fugro Ltd. |
Organisation | Fugro |
Country | Netherlands |
Sector | Private |
PI Contribution | The HUME project has spawned interest from other companies and in adjacent application areas. Specifically, Heriot-Watt University has been awarded a second Prosperity Partnership with Fugro Ltd, a world leader in geo-spatial intelligence. |
Collaborator Contribution | Prof. Yvan Petillot is PI and the total value of grant is £2.3M. |
Impact | Fugro Ltd. has recently transitioned to net-zero solutions for their marine operations using autonomous vehicles, remotely controlled from shore. This new grant will focus on intervention (manipulation) but the synergies with HUME are clear and the Chief Scientist of Fugro is now on the steering committee of HUME. |
Start Year | 2022 |
Description | 'Human-Machine Teaming' Talk by Bart Craenen at SeeByte Robotics Seminar Series |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | On May 25, 2022, Bart Craenen (Research Fellow working on HUME) gave a talk called 'Human-Machine Teaming', at Seebyte's Lunch with Robotics Seminar series. |
Year(s) Of Engagement Activity | 2022 |
Description | BBC Click Episode featuring Prof. Yvan Petillot and Joshua Roe |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | An episode of BBC Click episode went live on 21st May 2022 featuring HUME Co-I Prof. Yvan Petillot and National Robotarium Engineer Joshua Roe talking about maritime applications for autonomous systems. We are informed by the BBC that the programme is broadcast in more than 200 countries and attracts weekly viewing figures of 380 million. |
Year(s) Of Engagement Activity | 2022 |
Description | Furhat Conference Invited Speaker PI Helen Hastie |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Prof. Helen Hastie presented the Node, Human-robot teaming (HUME), ORCA and work with the Furhat. |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.youtube.com/watch?v=Uc-3t5gXz3g&t=2s |
Description | IWSDS Keynote: Prof. Helen Hastie |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Prof. Helen Hastie have a keynote talk where she discussed work on the Node, Human-robot teaming (HUME) and the ORCA hub |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.colips.org/conferences/iwsds2021/wp/ |
Description | Media Interview with Prof. Yvan Petillot, Joshua Roe and Dr. Jonatan Scharff Willners through the Agence France-Presse (AFP) |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Media attention was through the Agence France-Presse (AFP), the world's oldest news agency with bureaus in 151 countries. This took place on July 26th 2022, featuring Prof. Yvan Petillot as well as National Robotarium engineers Joshua Roe and Dr. Jonatan Scharff Willners, who have both contributed to the HUME project and supported the trials. The AFP media package secured international hits across Central and SE Asia, North America and Europe with an estimated ~900,000 views. Outlets included Taipei Times, News of Canada, France 24, Global Times Pakistan and Malay Mail. |
Year(s) Of Engagement Activity | 2022 |
Description | Organisation of a Robotics Collaborative Workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Other audiences |
Results and Impact | The workshop was with the Edinburgh Napier University, on the topic of robotics and collaboration and translation to industry/society. |
Year(s) Of Engagement Activity | 2023 |
Description | Organised Collaborative Robotics Workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Other audiences |
Results and Impact | The workshop was with the University of Edinburgh, on the topic of robotics and collaboration and translation to industry/society. |
Year(s) Of Engagement Activity | 2022 |
Description | PI Prof Helen Hastie talk about Robotics at National Robotarium in Scotsman's podcast |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | PI Prof Helen Hastie talk about the Node on Trust and human-robot collaboration in a Scotsman's podcast |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.scotsman.com/business/data-capital-podcast-how-data-and-ai-are-powering-robotics-revolut... |
Description | PI Prof. Helen Hastie- Living with Robots Panel Event |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Other audiences |
Results and Impact | This panel was part of the Edinburgh Futures Institute's series on Living well with Robots. This was live streamed and in-person. |
Year(s) Of Engagement Activity | 2023 |
URL | https://efi.ed.ac.uk/events/living-with-robots/ |
Description | Prof. Yvan Petillot - Guest Speaker in an Engineering Matters Live Episode |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | Prof. Yvan Petillot was invited to take part as a guest speaker in a Engineering Matters live episode in January 2023. The episode included a live discussion on the topic of The Revolution in the North Sea. This was a live recording that took place at the Royal College of Physicians in Edinburgh, UK. |
Year(s) Of Engagement Activity | 2023 |
URL | https://engineeringmatters.reby.media/live/ |
Description | Project Presentation by Prof. Yvan Petillot at the 'AI for Good' Seminar Series |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Prof. Yvan Petillot presented at the 'AI for Good' seminar series in June, 2022, on the future of robotics for offshore renewable inspection and intervention. |
Year(s) Of Engagement Activity | 2022 |
URL | https://aiforgood.itu.int/event/towards-the-future-of-the-energy-industry-with-intelligent-robots/ |
Description | Project Presentation by SeeByte at the European Marine Robotics & Applications Workshop (EMRA) 2022, Southampton, UK |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | SeeByte was invited to present the HUME project at the EMRA 2022 conference in Southampton, in June 2022. |
Year(s) Of Engagement Activity | 2022 |
URL | https://noc-events.co.uk/emra-2022 |
Description | Project Presentation by SeeByte at the MOOS-DAWG'22 Conference, Boston, USA |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Industry PI, Dr Andrea Munafo presented at the MOOS-DAWG'22 Conference in Boston, in August 2022, where he discussed the HUME project. |
Year(s) Of Engagement Activity | 2022 |
URL | https://oceanai.mit.edu/moos-dawg/pmwiki/pmwiki.php?n=Main.HomePage |
Description | Project Presentation by SeeByte during the Office of Naval Research (ONR) Virtual Programme Review |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Industry PI, Dr Andrea Munafo discussed some of the HUME results during the Office of Naval Research (ONR) Virtual Programme Review working group, in October 2022. |
Year(s) Of Engagement Activity | 2022 |
Description | SCRITA Workshop Invited Speaker- PI Helen Hastie |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | IEEE Ro-man Workshop on Trust, Acceptance and Social Cues |
Year(s) Of Engagement Activity | 2021 |
URL | http://scrita.herts.ac.uk/2021/ |
Description | SRPe Conference Presentation |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Conference showcasing Scottish Regional Partnership in Engineering (SRPe). Prof. Hastie's talk focused on equality and diversity in Robotics. |
Year(s) Of Engagement Activity | 2023 |
Description | STEM Event Presentation |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Undergraduate students |
Results and Impact | This was a Christmas event to celebrate Women in STEM, in particular robotics. |
Year(s) Of Engagement Activity | 2022 |
Description | System Demonstration |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Industry/Business |
Results and Impact | A major demonstration was completed in December 2023, involving the participation of all project partners and all work packages, showcasing project-wide work for the industrial partner SeeByte. Feedback from the demo has helped direct the research and development planned for the final year of the project. |
Year(s) Of Engagement Activity | 2023 |
Description | The Scotsman Newspaper Article written by PI Helen Hastie |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Newspaper article in 'The Scotsman' written by PI Helen Hastie on: 'Why teaching robots and humans to trust each other is essential'. |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.scotsman.com/news/opinion/columnists/why-teaching-robots-and-humans-to-trust-each-other-... |