Tactile superresolution sensing

Lead Research Organisation: University of Bristol
Department Name: Engineering Mathematics and Technology

Abstract

Superresolution encompasses a range of techniques for transcending the resolution limit of a sensor and earned the 2014 Nobel Prize in Chemistry (for superresolved fluorescence microscopy). Superresolution is analogous to biological hyperacuity of vision and touch where the discrimination is finer than the spacing between sensory receptors. Superresolution research in visual imaging has impacted science from cell biology to medical scanning 'in ways unthinkable in the mid-90s' (Editorial, Nature 2009).

The success of this proposal will enable the widespread uptake of superresolution techniques in the domain of artificial tactile sensing, potentially impacting multiple application areas across robotics from autonomous quality control in manufacturing to sensorized grippers for autonomous manipulation to sensorized prosthetic hands and medical probes in healthcare.

Proposed research

More specifically, the development of robust and accurate artificial touch is required for autonomous robotic systems to interact physically with complex environments, underlying the future robotization of broad areas of manufacturing, food production, healthcare and assisted living that presently rely on human labour. Currently, there are many designs for tactile sensors and various methodologies for perception, from which general principles are emerging, such as taking inspiration from human touch (Dahiya et al 2012), using statistical approaches to capture sensor and environment uncertainty and combining tactile sensor control and perception (Prescott et al 2012).

All application areas of robot touch are currently limited by the capabilities of tactile sensors. This first grant proposal aims to demonstrate that tactile superresolution can radically improve tactile sensor performance and thus potentially impact all areas of robotics involving physical interaction with complex environments. Visual superresolution has revolutionised the life sciences by enabling the imaging of nanoscale features within cells. Tactile superresolution has the potential to drive a step-change in tactile robotics, with applications from quality control and autonomous manipulators in manufacturing (Yousef et al 2011) to sensorized prosthetics and probes in healthcare.

Proposed initial application domain

Currently, across the entire automobile industry, gap and flush quality controls are made manually by human operators using their hands to check the alignment between vehicle parts. Experts in the industry have informed me that human hands are used because modern vision-based measuring technologies (such as laser scanners) do not robustly detect sub-millimetre misalignments between parts of differing reflectivity and refractivity. An automated system using robot touch would be more reliable, enable traceability of defects, and move production towards a fully automated paradigm.

The proposed research will culminate in a pilot study demonstrating that tactile superresolution will enable readily available tactile sensors to make gap and flush measurements of the requisite sub-millimetre tolerance and how the sensors should be controlled during the tactile perception task. This will constitute a first step towards building a consortium between academic and industrial partners to develop a fully working prototype for test installation on a production line.

Planned Impact

Tactile superresolution is a key enabling technology for existing tactile sensors to operate sensitivities sufficient for deployment in application areas such as healthcare technologies and manufacturing the future. The success of this proposal would have wide impact across Engineering, by raising awareness that superresolution can have many diverse applications, rather than being confined to conventional imaging applications such as microscopy in the life sciences.

Industry: One of the main beneficiaries of the proposed tactile superresolution research will be the automotive manufacturing sector where quality control of the finished product is of primary concern. In the UK, car production presently accounts for 11% of all UK exports, and over 1.5 million vehicles per year. I will utilize my contacts with Peugeot-Citroen PSA (who are supporting this proposal) and Centro Ricerche Fiat to disseminate the research findings and also submit a proposal for EU funding to develop the concept into a prototype for test installation on a production line. All results will be published to maximally disseminate the implications for industry. I will also utilize the University of Bristol's strong links with the UK transport manufacturing sector including Airbus (Filton site) and Rolls Royce, who are both based in Bristol, to build a complementary UK-based consortium.

Society: The success of this proposal will enable the widespread uptake of superresolution techniques in the domain of artificial tactile sensing applying to multiple application domains with significant societal impact. According to an influential recent report: 'robotics is the fastest growing industry in the world, and poised to become the largest industry in the next decade.' I will contribute to this growth by enabling robust and accurate artificial touch that is needed for autonomous robotic systems to interact physically with complex environments, underlying the future robotization of broad areas of manufacturing, food production, healthcare and assisted living that presently rely on human labour.

People: Output of trained people will also be an important outcome of the project. In addition to furthering the career of the named RA (Dr Martinez-Hernandez), I will involve BEng, MEng, MSc and PhD students in the proposed research, allowing them to develop skills in utilizing advanced robotics with application to industry. I am well-placed to enable the student participation as Program Director for the Bristol MSc in Robotics and as a member of the management panel for the Bristol CDT in Robotics. This training will help address the shortage of trained people in the fields of Robotics and Engineering in the UK. The equipment for demonstrating autonomous tactile quality control can also be used in University Open Days to stimulate school students' interest in Robotics and Engineering

Academic Career Impact

Given that the proposal is a First Grant Application, a key aspect of its impact is its ability to provide me with a framework for developing an independent research career. The success of the proposal would allow me to build up my research team (currently consisting of PhD and project students) to critical mass with the addition of an experienced postdoctoral researcher with whom I have already had a highly successful working relationship. A strong research team is needed to translate my research program into impactful outcomes.

The proposal research will impact the career of the Dr Martinez-Hernandez, who I supervised before moving to the University of Bristol. We could continue our successful teamwork that resulted in many publications in highly-ranked conferences and journals and laid the foundations for this proposal. I will also ensure that Dr Martinez-Hernandez makes full use of the training opportunities offered by the University of Bristol's Staff Development team, enabling him to become an independent researcher.

Publications

10 25 50

publication icon
Ward-Cherrier B (2016) Tactile Manipulation With a TacThumb Integrated on the Open-Hand M2 Gripper in IEEE Robotics and Automation Letters

publication icon
Lepora N (2016) Biomimetic Active Touch with Fingertips and Whiskers in IEEE Transactions on Haptics

publication icon
Lepora N (2016) Tactile Quality Control With Biomimetic Active Touch in IEEE Robotics and Automation Letters

publication icon
Ward-Cherrier B (2017) Model-Free Precise in-Hand Manipulation with a 3D-Printed Tactile Gripper in IEEE Robotics and Automation Letters

publication icon
Martinez-Hernandez U (2017) Active sensorimotor control for tactile exploration in Robotics and Autonomous Systems

publication icon
Ward-Cherrier B (2017) Exploiting Sensor Symmetry for Generalized Tactile Perception in Biomimetic Touch in IEEE Robotics and Automation Letters

 
Description 1) We demonstrate that an artificial tactile system offers a solution to autonomous quality inspection, using a biomimetic tactile fingertip mounted as end-effector on an industrial arm. The study considers a task of gap width inspection suitable for judging parts alignment, although the methods apply generally. We conclude that an artificial tactile system of the type here offers an ideal solution to automated quality control on the production line.
Reference: N. Lepora, B. Ward-Cherrier. Tactile quality control with biomimetic active touch. IEEE Robotics and Automation Letters, 2016.
2) Although superresolution has been studied to huge impact in visual imaging, it is relatively unexplored in tactile robotics. Here we demonstrate a novel optical sensor design (the TacTip) capable of achieving 40-fold localization superresolution to 0.1mm accuracy compared with a 4mm resolution between tactile elements. The attained superresolution is comparable to the best perceptual hyperacuity in humans.
Reference: N. Lepora and B. Ward-Cherrier. Superresolution with an optical tactile sensor In: Intelligent Robots and Systems (IROS): 2686-2691, IEEE/RSJ, 2015.
3) Tactile manipulation will be essential for automating industrial and service tasks currently done by humans. Here we present the TacThumb (Tactile Thumb): a cheap, robust, 3d-printed optical tactile sensor integrated on the Yale GrabLab model M2 gripper. This model-free approach gives a demonstration of basic tactile manipulation without the need for a kinematic model of the hand, in a manner that should generalize to other tactile manipulation tasks.
Reference: B. Ward-Cherrier, L. Cramphorn, N. Lepora. Tactile manipulation with a TacThumb integrated on the Open-Hand M2 Gripper. IEEE Robotics and Automation Letters, 2016
4) Tactile perception methods rely on creating mappings from tactile data to percepts. We introduce here a method to generalize tactile features. The generalization method leads to a strong reduction in the time needed to gather training data and only a moderate increase in classification error, and is particularly suited for multidimensional tactile data sampling in complex tasks such as tactile manipulation.
Reference: B. Ward-Cherrier, L. Cramphorn, N. Lepora. Exploiting sensor symmetry for generalized tactile perception in biomimetic touch. IEEE Robotics and Automation Letters, 2017
5) A key unsolved problem in tactile robotics is how to combine tactile perception and control to interact robustly and intelligently with the surroundings. Our approach brings together active perception and haptic exploration as instantiations of a common active touch algorithm, and has potential to generalize to more complex tasks requiring the flexibility and robustness of human touch.
Reference: N. Lepora, K. Aquilina, L. Cramphorn,. Exploratory Tactile Servoing With Active Touch. IEEE Robotics and Automation Letters, 2017
6) The fingerprint is a morphological aspect of the human fingertip that has interesting implications for our sense of touch. Here, we endow a biomimetic sensor with both papillary ridges (fingerprint) and a dermal stiffness contrast (stiffer intermediate ridges), and assess the impact on localization perception accuracy. This supports the theories that the fingerprint aids the classification of edges and smaller spatial scales, and demonstrates that the addition of a fingerprint to an artificial tactile sensors improves its acuity.
Reference: L. Cramphorn, B. Ward-Cherrier, N. Lepora. Addition of a Biomimetic Fingerprint on an Artificial Fingertip Enhances Tactile Spatial Acuity. IEEE Robotics and Automation Letters, 2017
7) In this paper, we present a novel and robust Bayesian approach for autonomous active exploration of unknown objects using tactile perception and sensorimotor control. Our work demonstrates that active perception has the potential to enable robots to perform robust autonomous tactile exploration in natural environments.
Reference: U. Martinez-Hernandez, T. Dodd, M. Evans, T., N. Lepora. Active sensorimotor control for tactile exploration. Robotics and Autonomous Systems 2017
8) This study provides a synthetic viewpoint that compares, contrasts, and draws commonalities for biomimetic perception over a range of tactile sensors and tactile stimuli. The active perception consistently reaches superresolved accuracy (hyperacuity) finer than the spacing between tactile elements. Biomimetic active touch thus offers a common approach for biomimetic tactile sensors to accurately and robustly characterize and explore non-trivial, uncertain environments.
Reference: N. Lepora. Biomimetic Active Touch with Fingertips and Whiskers. IEEE Transactions on Haptics 2016
9) Tactile manipulation is the ability to control objects in real-time using the sense of touch. Here we examine tactile manipulation from the perspective of active touch with a biomimetic tactile sensor, which combines tactile perception with control of sensor location. In general, it appears there is a trade-off between the responsiveness to unknown change and manipulation accuracy, which must be set appropriately for each task.
Reference: L. Cramphorn, B. Ward-Cherrier, N. Lepora. Tactile manipulation with biomimetic active touch. IEEE ICRA, 2016
Exploitation Route Superrresolution methods are relevant to any application of tactile sensing, as they give a method to improve the accuracy of tactile sensors. Applications range from measuring devices with touch, to application in grippers for picking up objects or manipulating objects in-hand. We demonstrated specifically in this grant how tactile superresolution applies directly to the automotive industry.
Sectors Digital/Communication/Information Technologies (including Software),Manufacturing, including Industrial Biotechology

URL http://www.lepora.com
 
Description 2017-2022) The Science Museum (London) is displaying a 3d printed tactile robot hand developed under this award as part of a robotics exhibition and in their permanent collection. This exhibition is still touring the country. Over 186,000 people saw the exhibition in London, and more than 66,000 in Manchester, making it the Museum of Science and Industry's most successful paid-for exhibition by a long way. 2020) The developments from this award (tactile superresolution, used to control a robot fingertip on the end of an industrial arm) has been developed since to interact fully with complex 3D environments, such as delicately feeling over the contours of a (model) face. Example video (Univesity of Bristol Engineering feed): https://twitter.com/UoBris_Innov/status/1233068613982134272?s=20 This has led to several public-interest broadcasts and stories including - 5-minute sequence on BBC4 Secrets of Skin Series (episode 6) on the relation between human and robot touch, https://www.bbc.co.uk/iplayer/episode/m000cf26/secrets-of-skin-series-1-6-sensing (15-20 minutes in) - Article on BBC online 'why your work colleague could be a robot' https://www.bbc.co.uk/news/business-51442445 2020) The National Composites Centre is currently trialling the technology developed in this award for defect detection in composite layup, as a core funded project supported by their stakeholders, such as Rolls-Royce and AirBus. 2019) This technology is also being trialled for the use of testing Ultrasound Haptic Displays, by the Bristol-based company Ultraleap (150 employees, £60M+ investor funding)
Sector Aerospace, Defence and Marine,Education,Manufacturing, including Industrial Biotechology,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic

 
Description Engineering Capital Equipment / Refurbishment/ Investment
Amount £22,000 (GBP)
Organisation University of Bristol 
Sector Academic/University
Country United Kingdom
Start 12/2015 
End 05/2016
 
Description Impact Acceleration Award
Amount £20,000 (GBP)
Organisation University of Bristol 
Sector Academic/University
Country United Kingdom
Start 09/2016 
End 03/2017
 
Description Leverhulme Leadership Award
Amount £1,000,000 (GBP)
Funding ID RL-2016-039 
Organisation The Leverhulme Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 04/2017 
End 03/2022
 
Description Pump priming
Amount £5,000 (GBP)
Organisation University of Bristol 
Sector Academic/University
Country United Kingdom
Start 11/2014 
End 06/2015
 
Title Addition of a biomimetic fingerprint on an artificial fingertip enhances tactile spatial acuity 
Description The following data was obtained from 3 variants of the TacTip (with fingerprint, with fingerprint and cores, without fingerprint), an optical tactile sensor integrated on a 6-dof robotic arm The sensor performed localization on 9 stimuli with varying spatial frequency over a 30 mm range. The data was used to demonstrate the effect of fingerprints on tactile spatial perception. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact This dataset was collected as part of the research for the paper: "Addition of a biomimetic fingerprint on an artificial fingertip enhances tactile spatial acuity". 
URL https://data.bris.ac.uk/data/dataset/2nky5v0esi6mu2pza3kncbb3eq
 
Title Dual-modal tactile perception and exploration 
Description EPSRC grant on tactile superresolution sensing (EP/M02993X/1) and Leverhulme Grant on 'Biomimetic Forebrain for Robot Touch' (RL-2016-39). Tactile sensing is required for human-like control with robotic manipulators. Multimodality is an essential component for these tactile sensors, for robots to achieve both the perceptual accuracy required for precise control, as well as the robustness to maintain a stable grasp without causing damage to the object or the robot itself. In this study, we present a cheap, 3D-printed, compliant, dual-modal, optical tactile sensor which is capable of both high (temporal) speed sensing, analogous to pain reception in humans and high (spatial) resolution sensing, analogous to the sensing provided by Merkel cell complexes in the human fingertip. We apply three tasks designed to test the sensing capabilities in both modalities; i) a depth modulation task, where a robot is required to follow a target trajectory using the high-speed modality; ii) an off-line high-resolution perception task, where the sensor perceives angle and radial position relative to an object edge; and iii) a tactile exploration task, where the robot uses the high-resolution modality to perceive an edge and subsequently follow the object contour. The robot is capable of modulating contact depth using the high-speed mode, attains a high level of accuracy in the perception task and accurate control using the high-resolution mode. The control method is successfully applied to an unseen object at an arbitrary depth with the use of both high-speed and high-resolution modalities in combination. 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? Yes  
 
Title Exploiting sensor symmetry for generalized tactile perception in biomimetic touch 
Description The following data was obtained from the TacTip, an optical tactile sensor integrated on a 6-dof robotic arm. Data was collected for edge orientation in 1 degree increments over a 360 degree range, and for contour following experiments location and identity data was collected over 360 deg (5 degree increments) and a 20 mm range (1 mm increments) across the edge. Data was then generalized using sensor symmetry from the first 30 degrees of collected data, and used to demonstrate contour following with the TacTip using generalized data. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact This dataset was collected as part of the research for the paper: "Exploiting sensor symmetry for generalized tactile perception in biomimetic touch". 
URL https://data.bris.ac.uk/data/dataset/vq6mpgxfyb8h2x0s6k85yu62r
 
Title Superresolution with an optical tactile sensor 
Description The following data was obtained from the TacTip, an optical tactile sensor while tapping vertically downwards on a 40mm diameter hemicylinder, at 4000 locations separated horizontally by 0.01mm. The data was used in a study exploring superresolution with the TacTip sensor. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact This dataset was collected as part of the research for the paper: "Superresolution with an optical tactile sensor". 
URL https://data.bris.ac.uk/data/dataset/1dusr7ea0y2hf11zq8cfbehd7r
 
Title Tactile manipulation with a TacThumb integrated on the Open-Hand M2 gripper 
Description The following data was obtained from the TacThumb, an optical tactile sensor integrated on a robotic gripper. The hand rolled a 25mm cylinder along the sensor in 0.5mm increments over a 20mm range. The data was used to demonstrate tactile manipulation in the open-hand M2 gripper 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact This dataset was collected as part of the research for the paper: "Tactile manipulation with a TacThumb integrated on the Open-Hand M2 gripper". 
URL https://data.bris.ac.uk/data/dataset/14tradpvkkd681344nss1lxvbz
 
Title Tactile manipulation with biomimetic active touch 
Description The following data was obtained from the TacTip, an optical tactile sensor while rolling a 25mm cylinder horizontally at 800 locations separated horizontally by 0.1mm. The data was used in a study demonstrating tactile manipulation with the TacTip mounted on an ABB industrial robot arm. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact This dataset was collected as part of the research for the paper: "Tactile manipulation with biomimetic active touch". 
URL https://data.bris.ac.uk/data/dataset/x1xiof2r2xa315nlad33ms7ow
 
Title Tactile quality control with biomimetic active perception 
Description The following data was obtained from the TacTip, an optical tactile sensor while tapping vertically downwards on a series of gaps cut in a horizontal perspex sheet. Gaps go from 0.25mm to 5mm in 0.25mm steps and are each tapped 110 times at depths ranging over 5.5mm. The data was used in a study investigating tactile quality control with the TacTip sensor. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact This dataset was collected as part of the research for the paper: "Tactile quality control with biomimetic active touch". 
URL https://data.bris.ac.uk/data/dataset/ejsgfgl8qrtq1gn8g43m65onq
 
Title Tactile quality control with biomimetic active perception 
Description The following data was obtained from the TacTip, an optical tactile sensor while tapping vertically downwards on a series of gaps cut in a horizontal perspex sheet. Gaps go from 0.25mm to 5mm in 0.25mm steps and are each tapped 110 times at depths ranging over 5.5mm. The data was used in a study investigating tactile quality control with the TacTip sensor. 10.1109/LRA.2016.2524071 
Type Of Material Database/Collection of data 
Year Produced 2016 
Provided To Others? Yes  
 
Title The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies 
Description EPSRC grant on tactile superresolution sensing (EP/M02993X/1) and Leadership Award from the Leverhulme Trust on 'A biomimetic forebrain for robot touch' (RL-2016-39). The following data was obtained from the TacTip, TacTip-M2, TacTip-GR2 and TacCylinder sensors, optical tactile sensors mounted on an ABBIRB120 robotic arm. The sensors rolled a cylinder of diameter 20 mm along a one-dimensional trajectory in 720 increments. The data was separated into 72 locations and used to validate the performance of the suite of TacTip tactile sensors. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
 
Title VIL: Early stage database sequences (obsolete) 
Description This dataset was uploaded as a test case only and has been superseded by DOI: 10.5523/bris.1oio9r9fm8byb1ia7mtv0sct0t. The dataset contains video sequences generated by the Visual Information Laboratory for encoding purposes, shot in 4K 120fps using a RED Epic camera. The sequences are 10secs each and are available in .mov format. 
Type Of Material Database/Collection of data 
Year Produced 2014 
Provided To Others? Yes  
 
Title Water-in-water emulsions 
Description Data supporting publication in J Colloid Interface Science by Ganley et al. on water-in-water emulsions 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? Yes  
 
Description Commercialisation - Neuromorphic tactile sensing (Inilabs) 
Organisation iniLabs Ltd
Country Switzerland 
Sector Private 
PI Contribution Development of a neuromorphic tactile sensor, using event-based coding for the tactile output based on the dynamic vision sensor developed by inilabs.
Collaborator Contribution Expertise on event-based vision sensors and help with commercialisation.
Impact Partnership still underway and covered by a NDA.
Start Year 2016
 
Description Commercialisation - Shadow Robotics 
Organisation Shadow Robot Company
Country United Kingdom 
Sector Private 
PI Contribution Customisation of Shadow Smart Grasping Systems to integrate a sense of touch.
Collaborator Contribution Donation and expertise in their Smart Grasping System
Impact Customization of their commercial prototype Smart Grasping System to have an integrated sense of touch
Start Year 2017
 
Description Press release on winning Harvard softrobotics competition 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Press release on the Tactile Robotics Team won first prize in the research category in the 2016 Soft Robotics Competition, from over 80 entries internationally. Our entry was on an open source 3d-printed tactile fingertip.
Led to a few interviews and several media articles internationally.
Year(s) Of Engagement Activity 2017
URL http://www.bristol.ac.uk/news/2017/january/tactip.html
 
Description donation to ROBOTS exhibiton Science Museum, London 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Donated output of research (3D-printed tactile sensor and robot hand) to the ROBOTS exhibition at the Science Museum, London
Year(s) Of Engagement Activity 2017
URL https://beta.sciencemuseum.org.uk/robots/