Morphological computation of perception and action

Lead Research Organisation: Imperial College London
Department Name: Design Engineering (Dyson School)

Abstract

Living beings share the same embodiment for sensing and action. For instance, the spindle sensors that provide the feeling of a joint angle and speed are embedded on the muscles that actuate this joint. The tendon sensors that provide the feeling of force too are directly involved in actuation of the joint. Do the function of these sensors change when the muscles are activated to take action? Does the co-activation of antagonistic muscles play a role not only in actuation, but also in perception? This project will investigate these questions through targeted experiments with human participants and controllable stiffness soft robots that provide greater access to internal variables.

Recent experiments we have conducted on localising hard nodules in soft tissues using soft robotic probes have shown that tuning the stiffness of the probe can maximise information gain of perceiving the hard nodule. We have also noticed that human participants use distinct force-velocity modulation strategies in the same task of localising a hard nodule in a soft tissue using the index finger. This raises the question as to whether we can find quantitative criteria to control the internal impedance of a soft robotic probe to maximise the efficacy of manipulating a soft object to perceive its hidden properties like in physical examination of a patient's abdomen.

In this project, we will thus use carefully designed probing tasks done by both human participants and a soft robotic probe with controllable stiffness to access various levels of measurable information such as muscle co-contraction, change of speed and force, to test several hypotheses about the role of internal impedance in perception and action. Finally, we will use a human-robot collaborative physical examination task to test the effectiveness of a new soft robotic probe with controllable stiffness together with its stiffness and behaviour control algorithms. We will design and fabricate the novel soft robotic probe so that we can control the stiffness of its soft tissue in which sensors will be embedded to obtain embodied haptic perception. We will also design and fabricate a novel soft abdomen phantom with controllable stiffness internal organs to conduct palpation experiments. The innovation process of the above two designs - the novel probe and the abdomen phantom - will be done in collaboration with three leading industrial partners in the respective areas. The new insights will make a paradigm shift in the way we design soft robots that can share the controllable stiffness embodiment for both perception and action in a number of applications like remote medical interventions, robotic proxies in shopping, disaster response, games, museums, security screening, and manufacturing.

Planned Impact

Academic impact: We will make academic impacts in three major areas in addition to extended impacts in groups of identified areas. First the findings in morphological computation (how the mechanical circuits of the embodiment contributes to the computation of perception and action, and their mapping from one to another) will shed new light on how the stiffness of a shared embodiment between perception and action can help to improve information gain in perception while improving the robustness of action. This will take research in soft robotics in a new path, because so far the attention has predominantly being on the role of embodiment in action. We will also contribute new technologies in soft grippers and probes that can regulate the stiffness of the embodiment to suit the task specific perception and action requirements. This project will also make new findings in human motor control related to the common mediation of muscle co-contraction in perception and action. So far there has been a heavy emphasis on the central nervous system as a direct mediator of perception and action, and the computational motor control community has given full credit of motor control to the brain. This project will challenge that norm with experiments done to find the role of indirect involvement of the central nervous system to regulate perception and action through joint impedance control.

Other scientific engagements: In addition to the main areas of scientific contributions, we will bring together a diverse group of scientists in biology and soft robotics to explore questions about the possible projections from neural based computational plasticity to regulate a common embodiment to improve perception and action on long-term morphological adaptation to reduce the burden on the central nervous system to perform repetitive computational tasks. Moreover, the engagement with the telecommunication research community will help us to make an impact on the future "haptic internet" where we will have efficient communication protocols to perform remote physical examinations with real-time haptic feedback with guarantees on safety.

Industrial and clinical impact: There is an increasing demand for soft robotic grippers and probes in surgery, agriculture, remote healthcare, museums, tele-shopping, games, and aviation. The scientific contributions to introduce new ways to improve perception and action through a shared controllable stiffness embodiments together with novel ways to design and fabricate soft grippers/probes for physical examination of soft tissue will make a disruptive shift in the practice of the soft robotics industry.

Social impact: The project will contribute to a broad social discussion on how soft robots can be used in our daily lives as well as to understand our own bodies through engaging students and robot enthusiasts in scientific exhibitions, information on our websites, and meetings in the London Robotics Network.

Impact on students and RAs: The postgraduate students associated with this project and the directly employed RAs will have the opportunity to teach and conduct public engagement workshops. We have budgeted for their travel to major international conferences to promote their visibility and to facilitate networking. Following our previous experiences, we have planned to integrate technological innovations in this project to formal teaching practice of robotics, neuroscience, and general practitioner training.

Open access data: Following positive experiences of releasing open access data based on previous EPSRC funded projects, we will release anonymised physical examination behavioural data with proper guides to use them for research purposes. This will engage a broad academic community interested in related research areas.

Publications

10 25 50
 
Description 1) Stiffness control in a soft robotic probe plays a key role in the haptic information gain by its tendon sensors. This also explains why humans control the finger stiffness during soft tissue examination. 2) Haptic perception and muscle activation are inter-linked. In the case of haptic based guiders, the same amount of leftward and rightward perturbations won't be felt equally by the follower. In the case of medical examination of patients, these insights into how action and perception are coupled will allow us to train medical students better. 5. The tissue abnormality detection process involves first depending mainly on fingertip tactile sensors for search area reduction and then using tendon level haptic feedback to explore details such as the size and the depth of an abnormality.
6. Granular jamming with positive pressure is a viable method to both produce tumor conditions in a robotic abdominal phantom and to use them as sensors to measure palpation forces and positions.

We disseminated the outcomes in a series of keynotes and invited talks for an international audience:

23/09/2019: UK/US Bio-inspired Unmanned Autonomous Systems (BioUAS), Plenary talk by Thrishantha Nanayakkara, titled "The brain is lazy" highlighting the clever ways it takes to condition the body to simplify autonomous control, London, UK.

27/06/2019: Robotics Science and Systems (RSS) 2019, Plenary workshop talk by Thrishantha Nanayakkara on "Embodied computation of perception and action for fast uncertainty reduction", Freiburg, Germany

14/04/2019: IEEE RoboSoft 2019: Plenary workshop talk by Thrishantha Nanayakkara at IEEE RoboSoft 2019 on "Towards the nature of information processing in soft machines", Seoul, South Korea

14/04/2019: IEEE RoboSoft 2019: Workshop organizer and workshop talk by Thrishantha Nanayakkara at IEEE RoboSoft 2019 on "Morphological Computation Through Physical Adaptation of Soft Robots", Seoul, South Korea

24/04/2018: IEEE RoboSoft 2018, Plenary workshop talk by Thrishantha Nanayakkara at IEEE RoboSoft 2018 on "Haptic information gain in the impedance domain", Levorno, Italy

23/05/2018: IEEE ICRA2018, Plenary workshop talk by Thrishantha Nanayakkara at IEEE ICRA2018 on "Brain is the last resort", Brisbane, Australia

27/05/2018: IEEE ICRA2018, Invited workshop talk by Thrishantha Nanayakkara at IEEE ICRA2018 on "Active search for haptic information in the impedance domain", Brisbane, Australia

02/05/2018, University of Leeds, Keynote by Thrishantha Nanayakkara on "Surviving in an uncertain world with slow communication pathways", Leeds, UK

04/10/2018: RCGP-2018, Workshop keynote talk by Thrishantha Nanayakkara at Royal College of General Practitioners annual conference on "Soft Robotic Patients for Medical Education", Glasgow, UK

17/08/2018: Xuzhuo Central Hospital, Keynote talk by Thrishantha Nanayakkara at Xuzhou Central Hospital on "Soft Robotics for Rehabilitation", Xuzhou, China

23/08/2018: Jilin University, Keynote talk by Thrishantha Nanayakkara at International Workshop on Bioinspiration on "Morphological computation - The role of the physical circuits in the body in the computation of perception and action", Jilin, China

11/10/2017: IEEE PIMRC, Plenary keynote talk by Thrishantha Nanayakkara at IEEE PIMRC on "Survival in uncertain environments with slow communication pathways - evidence from morphological computation", Montreal, Canada

12/12/2017: IMechE Christmas event, Keynote talk by Thrishantha Nanayakkara at IMechE Christmas event - "Tinkering" on "If robots were to survive like living beings in uncertain environments", London, UK
18/12/2017: LIPI workshop, Keynote talk by Thrishantha Nanayakkara at Indonesia Institute of Science and technology on "Soft robotics for sustainable rainforest communities", Bandung, Indonesia

10/07/2017: 2nd UK manipulation workshop, Invited talk by Thrishantha Nanayakkara on "Surviving with slow communication pathways", London, UK

23/03/2017: ERU-2017, Co-organizer and speaker in the Workshop on "Haptics for Healthcare", Edinburgh, UK
Exploitation Route 1) Further develop the soft robotic patient with more functionality like breathing, controllable internal organs, and a face that can express pain, as a tool for medical training and to collect examination related data from general practitioners, 2) Extend the soft robotic probe for soft tissue palpation as a tool in the food industry in addition to healthcare applications.
Sectors Agriculture

Food and Drink

Communities and Social Services/Policy

Creative Economy

Digital/Communication/Information Technologies (including Software)

Education

Healthcare

Leisure Activities

including Sports

Recreation and Tourism

Culture

Heritage

Museums and Collections

URL https://thrish.org/publications
 
Description A soft robotic patient is being developed for medical training purposes. The first version was tested with General Practitioners on 22nd February 2018. We further did a workshop in the annual conference of the Royal College of General Practitioners held in Glasgow on 4th August 2018 to engage the broad physician community and they expressed much hope in a future functional robotic patient for training purposes. We have developed controllable soft organs that can present different physiological conditions to a practitioner during a training session. We are now conducting experiments to understand the role of facial expressions of pain together with local muscle tensions in response to pain as additional feedback to train physicians. The work led to further EPSRC funding under the RoboPatient grant. We have improved the sensing and actuation capabilities of the soft robotic internal organs to monitor physical examination behaviors of a trainee. Following up, I secured a UKRI Impact Acceleration award to launch a new startup on medical simulators.
First Year Of Impact 2024
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Electronics,Healthcare
Impact Types Societal

Economic

 
Description Written evidence with my Views on a New UK Funding Agency for Science and Technology - ARIA
Geographic Reach National 
Policy Influence Type Contribution to a national consultation/review
Impact It was directed at the launch of ARIA
URL https://committees.parliament.uk/writtenevidence/6850/pdf/
 
Title Data set for PONE-D-17-33558R2 
Description This is the data set for the PloS One paper: Giuseppe Cotugno, Jelizaveta Konstantinova, Kaspar Althoefer, Thrishantha Nanayakkara, Modelling the Structure of Object-Independent Human Affordances of Approaching to Grasp for Robotic Hands, accepted in Plos One, PONE-D-17-33558R2. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
 
Title Dataset for Conditioned haptic perception for 3D localization of nodules in soft tissue palpation with a variable stiffness probe 
Description These data are complementing the following publication:[1] N. Herzig, L. He, P. Maiolino, S-A Abad, and T. Nanayakkara, Conditioned Haptic Perception for 3D localization of Nodules in Soft Tissue Palpation with a Variable Stiffness Probe. PLoS One. DOI: 10.1371/journal.pone.0237379These data support our research on a Variable Stiffness Palpation Probe and its control strategy to palpate and detect the location of stiff inclusions in soft tissues.The folder contains a ReadMe file and a binary Matlab file. For more details about the content of the binary file and the data structure, please read the ReadMe file. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://figshare.shef.ac.uk/articles/dataset/Dataset_for_Conditioned_haptic_perception_for_3D_locali...
 
Description Collaboration with University of Oxford on AI based virtual patients 
Organisation University of Oxford
Department Oxford Hub
Country United Kingdom 
Sector Academic/University 
PI Contribution I coordinated a new EU grant proposal with partners from Oxford.
Collaborator Contribution Partners from the Institute of Biomedical Engineering, University of Oxford, contributed to a work package on AI based virtual patients as part of a new robotic patient to train medical students on physical examination.
Impact Submitted an EU grant proposal for review.
Start Year 2023
 
Description Collaboration with the medical partners at the Hammersmith Hospital 
Organisation Hammersmith Hospital
Country United Kingdom 
Sector Hospitals 
PI Contribution We provided the new robotic patient to be tested on medical students, and conducted experiments along with clinical teachers.
Collaborator Contribution The clinical collaborators at the Hammersmith hospital helped to recruit medical students for human participant experiments, and gave valuable clinical insights to test the new robotic patient to train physical examination skills in medical students. They also partnered to write a new EU grant proposal.
Impact We have submitted a new EU grant proposal with 2 more EU partners. The clinical collaborators from Hammersmith hospital made valuable contributions to write a work package.
Start Year 2021
 
Description Collaboration with the medical simulation centre, Faculty of Medicine of Masaryk University 
Organisation Masaryk University
Country Czech Republic 
Sector Academic/University 
PI Contribution I coordinated an EU grant proposal with partners from the Faculty of Medicine of Masaryk University
Collaborator Contribution The partners from Faculty of Medicine of Masaryk University invited me for a visit to see the new medical simulation centre and made valuable contributions to a work package in a new EU grant proposal.
Impact A new EU grant proposal submitted for review in February 2024.
Start Year 2023
 
Description 2017 Cambridge Workshop on Soft Robotics, Morphological Computation, and Embodied Intelligence 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact This was a 3-day resident workshop to brainstorm ideas and latest advancements in Soft Robotics, Morphological Computation, and Embodied Intelligence in several laboratories in UK. PhD students in my lab got an opportunity to present their work to an audience of key research labs in this area in UK and share thoughts.
Year(s) Of Engagement Activity 2017
URL https://thrish.org/2017/2017-cambridge-workshop-on-soft-robotics-morphological-computation-and-embod...
 
Description A hands on testing session with General Practitioners 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact In this workshop, General Practitioners examined our soft robotic patient that could present illness conditions like liver abnormalities, and they experienced how the new wrist-mounted haptic feedback system can be used to feel the forces experienced by a remote robotic soft probe examining a soft tissue. A unique feature in this remote examination was that the doctors could control the stiffness of the remote robotic probe by turning a handle. They reported that the stiffness control of the remote soft robotic probe makes a difference in the way they feel a nodule in the remote soft tissue. Moreover, they gave valuable feedback to improve the soft robotic patient and the remote haptic feedback system.
Year(s) Of Engagement Activity 2018
URL https://twitter.com/thrishlab/status/966815438838665216
 
Description A keynote speech to the staffers of honourable MPs in the Sri Lankan parliament on "broad trends in AI and robotics" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I was invited to give a keynote on "broad trends in AI and robotics - how to prepare to benefit from the economic wave" to the senior staffers of the honourable MPs of the Sri Lankan parliament. I also published the speech on Youtube and Linkedin.
Year(s) Of Engagement Activity 2024
URL https://thrishantha.blogspot.com/2024/02/a-formula-to-ignite-youth-tech-start.html
 
Description BBC Radio 4 Bottom Line panel discussion on robots 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Are white collar workers under threat from the new robotic workforce? Manufacturing companies have been replacing production line workers with robots for years, but as the technology becomes more advanced, they are being increasingly used in healthcare, law firms and HR departments. So is the white collar worker under threat? Evan Davis talks to three guests, a roboticist, an Artificial Intelligence consultancy worker and the CEO of a hotel chain who is already using robots in his business.

GUESTS

Thrishantha Nanayakkara - Roboticist, Imperial College

Dr. Michael Chui - Partner, IT and innovation, McKinsey Global Institute

Hubert Viriot - CEO, Yotel.
Year(s) Of Engagement Activity 2017
URL http://www.bbc.co.uk/programmes/b08v8p14?ns_mchannel=social&
 
Description Chaired a keynote session on Morphological Computation at IEEE International Conference on Robotics and Automation (ICRA 2023) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I chaired a keynote session on Morphological Computation at ICRA23, the World's largest international conference in robotics. I mentioned in my Linkedin post that "The idea that physical structures can to a certain extent interact with their environment on their own without sensors or computers has been a fascinating area for me. Theo Janssen's walking sculptures on beaches is one example. The wind and the beach is all what the sculpture needs to walk like an animal. How Jamie Paik's modular origami inspired robots can scale up to exhibit complex behaviours was mind boggling. Fumiya Iida's robots printing their own extensions such as spoons to perform tasks better was fascinating."
Year(s) Of Engagement Activity 2023
URL https://www.linkedin.com/posts/thrishantha_i-enjoyed-chairing-the-morphological-computation-activity...
 
Description Creative Quarter 2017 at Imperial College South Kensington campus 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact Creative Quarter is a day of free events in South Kensington offering thousands of students between the ages of 13 and 19 the chance to explore work in the creative and STEM industries and careers. At Imperial students could find inspiration through interactive exhibits, demonstration lectures and a series of smaller career insight sessions where they get to meet some of the scientists working at Imperial and hear what it is like to be a science professional.
Year(s) Of Engagement Activity 2017
URL https://thrish.org/2017/thrishlab-at-imperial-creative-quarter-2017/2513
 
Description Harvard University, Invited talk titled A soft robotics approach to understand biological secrets of complex interactions 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Harvard University, Invited talk titled A soft robotics approach to understand biological secrets of complex interactions. This talk was given to roboticists at the John Paulson School of Engineering and Applied Sciences. There were special invitees from Harvard Kennedy School of Government too.
Year(s) Of Engagement Activity 2023
 
Description IIEEE IAS GlobConET 2023, Keynote speech titled A robotics approach to understand how the brain tunes the body to solve dynamic interaction problems 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact IEEE IAS GlobConET 2023, Keynote speech titled "A robotics approach to understand how the brain tunes the body to solve dynamic interaction problems". This was based on results from human participant experiments in the MOTION and robo-patient projects.
Year(s) Of Engagement Activity 2023
URL https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10149910
 
Description Imperial festival 2017 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact What started as a modest pilot project in 2012 to explore how Imperial College London might share its research with more people in new ways has now evolved into the College's flagship public event. In 2017 more than 20,000 public and alumni visitors descended on Imperial's South Kensington campus to enjoy the interactive stands, workshops, tours, talks and performances on offer.

Some lovely reviews from our friends in the media support our belief that the Festival has grown into a great weekend of fun and learning for all the family - with time Timeout London describing it as an "an equation for a good time", the Guardian saying it was "a fascinating event" and the Londonist observing that "whoever programmed this glorious beast must have used hyper-dimensional graph paper".
Year(s) Of Engagement Activity 2017
URL https://thrish.org/2017/our-exhibits-in-the-imperial-robotics-festival-2017/2344
 
Description Plenary Keynote speech at IEEE PIMRC-2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact IEEE PIMRC is said to be the largest IEEE mobile communications conference sponsored by top communications companies. Their purpose of inviting me was to know more about how living beings survive in uncertain environments that require fast responses with a central nervous system (CNS) known to have very slow communication fibres (a signal from fingertip to brain takes about 120msec). Most rapid responses cannot be explained using such slow communication pathways.

My talk using robotic experimental evidence on the role of fast bodily computers (morphological computers) that work in conjunction with the CNS helped to challenge the conventional notion that the slow CNS is entirely responsible for intelligent behavior. This aligned well with the latest trends in 5G communication where field experts argue that some new decentralized solutions are required to go beyond the current bottlenecks.
Year(s) Of Engagement Activity 2017
URL http://pimrc2017.ieee-pimrc.org/program/keynotes/
 
Description Talk at Science Museum Lates 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact In the science museum lates, adult general public gets a chance to interact with various invited speakers and exhibitors to discuss latest advancements in different fields. In this talk, I explained our latest advancements in the EPSRC funded MOTION project.
Year(s) Of Engagement Activity 2017
URL https://www.sciencemuseum.org.uk/see-and-do/lates