Human Centred Robotics for Next-generation Flexible Manufacturing

Lead Research Organisation: University of the West of England
Department Name: Faculty of Environment and Technology

Abstract

The proposed research falls within the field of Robotics and Artificial Intelligence Systems; an area that has enormous potential to provide greater levels of throughput, repeatability, productivity and the introduction of more complex tasks to be carried out in a robot collaboration environment within the UK Manufacturing Sector. The introduction of robots in the production process has undeniable benefits: each robot can replace several human operators, performing repetitive tasks. However, reprogramming and operating robots for production purposes can pose significant challenges for businesses, which could be potential barriers to automation and corporate expansion. For example, each time a robot undertakes a new task it must be reprogrammed. Reprogramming multiple robots can take up to several months and involve the services of a specialist robot programmer. This results in high running costs and suboptimal productivity creating a barrier to the adoption of the technology within the wider manufacturing sector. Similarly, colocation of robots with humans and other machines requires higher levels of cognition, perception and autonomy to assimilate with different user experiences and individual preferences, without interfering with operational schedules.
The research is aimed to address these issues and ensure that robotics are more widely adopted, with the intention of producing software and hardware toolkits that once commercially available will enhance efficiency, reduce costs and facilitate corporate expansion. To achieve this two key approaches will be investigated:
Demonstrable (WS1) - which will develop a new skill transfer interface to teach the robots through body posture, hand gesture and voice commands. It will include (a) a comprehensive human motor skills capture system based on fusion of both physical signals including motion and force and physiological signals of muscle internal activities; (b) a user friendly intuitive teaching interface integrating the skill capture system with mixed reality and voice control, (c) a holistic approach to capture and transfer manipulative skills of arm and hand as coordinated system; and (d) skill generalization mechanism for robots to perform new tasks without additional demonstration.
Collaborative (WS2) - to deliver an intelligent control system for cobots to achieve optimal human-robot cooperation, so that a human's flexibility and creativity can be efficiently integrated with a robot's accuracy and repeatability. This involves (a) a reliable and efficient gesture/posture/voice based communication channel for human co-workers to command the robots easily; (b) improved cobot cooperation skills by embedding human intent perception into robot's control actions, (c) learning strategies to capture individual human co-worker's motion/force pattern for a cobot to provide customized support, and (d) validation in commercially available cobots such as KUKA iiwa and UR5-CBR, together with 3-finger Robotiq gripper and 5-finger Wessling Robotic Hand.
The collective outcome will be innovative, user friendly, technology that permits existing members of the workforce to train robots to undertake new tasks - reducing the cost of outsourcing to one fifth and enabling reprogramming to be completed at a rate that is approximately ten times faster than previous methods. This will have notable economic benefits for distributors of the software and companies as end-users within the manufacturing sector. Not only will existing production lines be more cost effective and profitable but new markets (i.e. customisation and the delivery of new products) will be accessible because of the ability to swiftly reprogram robots for new tasks. Therefore, corporate expansion will be facilitated, via the adoption of digital technology (a priority area for the UK Government), ultimately bolstering the UK Economy.

Planned Impact

Digital technology, especially Robotics and Artificial Intelligence Systems, can orchestrate a transformational effect on industry. Robots are able to provide manufacturers with a plethora of benefits; guaranteeing consistency in repeated tasks, high throughput rates, operating in environments and conditions that may be considered unsafe for humans, performing with accuracy, minus the necessity for breaks, holidays and compliance with health and safety regulations in the workplace.

However, automation can be both costly and disruptive. The initial outlay for a robotic unit is further exacerbated by the cost of reprogramming and operational challenges. An industrial robot will require specialist programming and integration for every new task (for instance the control system of an industrial robotic arm will take >200 hours to reprogram) in order to meet the emergent customisation and flexibility of production requirements. This is usually outsourced or requires the recruitment of a full-time specialist which is both time consuming and expensive (particularly for SMEs). Furthermore, employees are required to adapt to the new technology and difficulties with user experiences notably underpin costly delays in production.

This research, undertaken as part of the Fellowship, will target these challenges, resulting in commercially available software that will lead to significant economic benefits for distributors of the software and, on a wider scale, financial sustainability and expansion opportunities for the companies within the Manufacturing Sector. Moreover, long-term, it will facilitate the UK's economic development through the uptake and application of transformational technology.

Commercial transformation of the research outputs and the knowledge gained through further industrial participation will follow the IP protection of commercially relevant results. This is expected to yield substantive revenue for the prospective co-developers . The software will be sold to end-users, organisations within the manufacturing domain, generating wealth creation through the availability of a new product.

Organisations within the Manufacturing Sector and their employees will benefit from reduced costs, increased productivity and by gaining access to new markets through customisation and flexibility of production. As opposed to outsourcing, the software and hardware toolkits will enable existing employees to reprogram the robots to undertake new tasks. This will not only be cost effective but will motivate potentially upskill members of the workforce. Additionally, the new technology will allow robotic reprogramming to be accomplished at a markedly faster rate, approximately ten times faster than existing methods, avoiding costly delays in production and potentially facilitating increased revenue via entry into new markets (i.e the production of new products, customisation). Notably, this would previously have been challenging for many SMEs, given the amount of time and expense to teach robots on the production line to undertake new tasks. Finally, the ability to customise individual user experiences will ultimately increase efficiency.

The UK economy can be strengthened from expansion in the manufacturing sector with the ability to produce ever-more customised products, reversing the cost advantage of low-wage economies, incentivizing companies to re-shore activities to the UK, relocating closer to their customer base. This has already been the focus of key industrial nations in the EU, China, Japan and the USA to bolster their economies. The research offers a pathway for organisations within the Manufacturing Sector, who have already automated, to increase profits and expand production. It will also serve as an incentive for organisations who have yet to purchase robotics, by mitigating concerns about additional costs and maintenance and highlighting a route to increased productivity.

Publications

10 25 50
 
Description -We have proposed novel robot grasping methods based on image processing and reinforcement learning
-We developed and tested deep neural network approach for robot dynamics calibration
-We investigated robot impedance control/compliant control in a hierarchinal manner
-We have developed neural learning enhanced variable admittance controllers for human-robot physical collaboration, such that human effort can be reduced while the task performance being enhanced.
-We have established a robot learning framework based on adaptive admittance control and have built generalizable motion models
-We studied cooperation of multiple manipulators for a common task, and exploited disturbance observer technique to deal with input saturation. We also investigated admittance based multi-robot cooperation in the presence of output constraints.
-We studied force observer and have successfully applied it for admittance control design for robot manipulator of dynamics uncertainties, which are compensated for by neural learning approach. We also considered the problem of actuator saturation, which usually happen such that robot may not be able to provide enough large torques for actuation.
-We investigated random forest-based image processing technique to facilitate robot grasping based on vision guidance, and designed an adaptive method of shape approximation using Gaussian mixture models for robot grasping.
-We proposed a new force estimation method using muscle electromyography and developed a multimodal sensory data encoding approach for robotic skill learning.
-We have also investigated deep neural network techniques, and have successfully applied it on anthropomorphic manipulators for human-like redundancy optimization, as well as tool identification and calibration for bilateral teleoperation.
-We have consider constraints in skill representation and developed composite movement primitives, and develop incremental learning of movement primitives using fuzzy logic.
-We have investigated human intention for trajectory on-line adaptation and developed bio-mimetic control and proposed an improvement of robot stiffness-adaptive skill primitive generalization for human-robot collaboration
Exploitation Route Multimodal skill learning techniques for robot have been further developed in tihs project. Learning human skills will be very useful for robotic scanning of non-standard objects in the non-destructive testing (NDT), and collaboration with TWI is ongoing to investiage human robot collaborative scanning. The human robot collaboratation techniques developed have also been employed and tested in robotic surgery experiment, and will be interesting to researchers in surgeric robotics. The skill learning and human robot collaboration approaches we developed can also potentially be used for robotized physical examination, e.g., ultrasound scanning of patient by a robot, with advanced compliant controller and dexterous manipulation skills.

Extensive studies made on physical human robot collaboration especially those on stiffness adaptation and trajectory adaptation are useful for human robot collaborative manufacturing. And constrained skill learning framework will help represent skills learnt in the manufacturing scenarios where workspace is not fully accessible.
Sectors Digital/Communication/Information Technologies (including Software)

Healthcare

Manufacturing

including Industrial Biotechology

 
Description As the corresponding co-chair, the fellow sponsored by this grant manages the technical activities of the Technical Committee on Collaborative Automation for Flexible Manufacturing (CAFM) of the IEEE Robotics and Automation Society, the organizer of the world's largest robotics conferences ICRA (International Conference on Robotics and Automation) and IROS (IEEE/RSJ International Conference on Intelligent Robots and Systems). The fellow has been appointed to the Associate Editor of these two major robotics conferences, in addition to his appointment of Associate Editorship of IEEE Transactions on Automation Science and Engineering (2020 -2023) and IEEE Transactions on Robotics (2020 - 2024). He is also appointed as a Senior Editor of IEEE Transactions on System, Man and Cybernetics: Systems. respectively. These enables the fellow to influence widely to academic, industrial and general public communities. The research outcomes of this fellowship have been well disseminated through not only publications but also the fellow's active involvement of conference organizations. For example, the fellow has served as General Chair of The 25th IEEE International Conference on Industrial Technology (ICIT), 25-27 March 2024, Bristol, UK and The 27th IEEE International Conference on Automation and Computing (ICAC), 1-3 September 2022, Bristol, UK, in addition to the program co-chair of the 19th UK Workshop on Computational Intelligence, and program chair of the 26th IEEE International Conference on Automation and Computing to be held in Portsmouth this September. These conferences create opportunities for researchers to exchange ideas with industrial participants and will promote public awareness of the importance and robotics and AI.
First Year Of Impact 2021
Sector Digital/Communication/Information Technologies (including Software),Healthcare,Manufacturing, including Industrial Biotechology
Impact Types Cultural

Societal

Economic

 
Title Optimization of Human-Robot Collaboration based on Adaptive Variable Admittance Control 
Description We developed a new method to optimize physical human robot interaction based on variable admittance control. By using measured electromyography to esimate human muscle activation, we adapt robot controller to reduce human effort and improve control performance simutaneously. 
Type Of Material Model of mechanisms or symptoms - human 
Year Produced 2020 
Provided To Others? Yes  
Impact This method can be very promising to improve efficiency of physical human robot collaboration. 
 
Description Collaboration with Politecnico di Milano 
Organisation Polytechnic University of Milan
Country Italy 
Sector Academic/University 
PI Contribution Our team informed colleagues at Milan about our advances in robot learning techniques, and discussed possible approaches to apply these learning algorithm on surgical robots.
Collaborator Contribution Colleagues at Milan implement our co-developed algorithms to control surgical robots.
Impact -W. Qi, H. Su, C. Yang, G. Ferrigno, E. De Momi, A. Aliverti, A Fast and Robust Deep Convolutional Neural Networks for Complex Human Activity Recognition using Smartphone, Sensors 19 (17), 3731, 2019. DOI: 10.3390/s19173731 -H. Su, W. Qi, C. Yang, A Aliverti,G. Ferrigno and E. D. Momi, Deep Neural Network Approach in Human-like Redundancy Optimization for Anthropomorphic Manipulators, IEEE Access, vol. 7, no. 1, DECEMBER 2019, pp. 124207-124216. DOI:10.1109/ACCESS.2019.2937380 -H. Su, C. Yang, H. Mdeihly, A. Rizzo, G. Ferrigno, and E. D. Momi, Neural Network Enhanced Robot Tool Identification and Calibration for Bilateral Teleoperation, IEEE ACCESS, vol. 7, no. 1, DECEMBER 2019, pp.122041-122051. DOI:10.1109/ACCESS.2019.2936334 -H. Su, C. Yang, G.Ferrigno, and E. D. Momi, Improved Human-Robot Collaborative Control of Redundant Robot for Teleoperated Minimally Invasive Surgery, RA-L and ICRA for the IEEE Robotics and Automation Letters (RA-L), vol. 4., no. 2, 2019, pp. 1447-1453. DOI: 10.1109/LRA.2019.2897145
Start Year 2019
 
Description TWI Technology Centre (Wales) 
Organisation TWI ltd
Country United Kingdom 
Sector Private 
PI Contribution Collaboration on human robot collaborative NDT process for non-standard objects to be scanned
Collaborator Contribution Informing us the experiences in robotised NDT Providing advice on NDT relatd technologies Promoting the joint reserach outcomes
Impact The collaboration is ongoing and is multi-disciplinary, involving control engineering, machine learning and computer vision.
Start Year 2019
 
Description ? International Organizing Committee Chair, The 12th International Conference on Intelligent Robotics and Applications (ICIRA), August 8-11, 2019 Shenyang, China. https://www.icira2019.org/ 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I was involved in the organization of the International Organizing Committee Chair of the 12th International Conference on Intelligent Robotics and Applications (ICIRA), August 8-11, 2019, at Shenyang, China.
Year(s) Of Engagement Activity 2019
 
Description Distinguished Lecture to Universiti Teknologi Malaysia 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact The fellow has been invited by Falcuty dean Prof. Dato' Ir. Dr. Mohammed Rafiq bin Dato' Abdul Kadir, to deliver a Distinguished Lecture to Faculty of Engineering,Universiti Teknologi Malaysia, on 24 Aug 2020. A few hundres of audience attended the talks and a large number of questions have been answered. The research outcomes of this fellowship have sparked great interest from the colleagues in Malaysia and promoted the idea of physical human robot collaboration of flexible manufacturing.
Year(s) Of Engagement Activity 2020
 
Description Invited and Special Session Chair of ICIEA2020 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The fellow was the Invited and Special Session Chair of the 15th IEEE Conferenceon on Industrial Electronics and Applications (ICIEA2020), 9-13 Nov, Kristiansand, Norway. This conference attracted hundreds of participants covering a large range of topics.
Year(s) Of Engagement Activity 2020
 
Description Organization Chair of ICIST 2020 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The fellow plays the role of Organization Chair for the 10th International Conference on Information Science and Technology (ICIST 2020), virtually hosted at Bath on 3-5 Sept 2020. Over one hundred participants joined the online conference and extensive discussions have been made among researchers.
Year(s) Of Engagement Activity 2020
 
Description Program co-chair of the 19th UK Workshop on Computational Intelligence, September 4-6, 2019, Portsmouth, UK 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact As the program co-chair, I jointly organized the 19th UK Workshop on Computational Intelligence, September 4-6, 2019, Portsmouth, UK, and co-edited a book titled Advances in Computational Intelligence Systems: Contributions Presented publised at Springer International Publishing. ISBN: 978-3-030-29933-0; DOI: 10.1007/978-3-030-29933-0
Year(s) Of Engagement Activity 2019
 
Description Publicity Chair of Ro-MAN 2020 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The fellow plays the role of Publicity Chair for the 29th IEEE International Conference on Robot and Human Interactive Communication (Ro-MAN) in Naples, Italy, 31th August to 4th September 2020. This is a very popular conference in the robotics communities and a few hundreds scholars attended the conference with extenstive discussions.
Year(s) Of Engagement Activity 2020
 
Description Review panel member of EPSRC HT Investigator-led Panel Meeting, 31 Jan 2019 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Supporters
Results and Impact I was invited to be a panel member of EPSRC HT Investigator-led Panel Meeting, 31 Jan 2019
Year(s) Of Engagement Activity 2019
URL https://gow.epsrc.ukri.org/NGBOViewPanel.aspx?PanelId=1-6ASLMJ