Sensory illusions using low-cost haptic devices to enhance spatial understanding for engineering design

Lead Research Organisation: University of Nottingham
Department Name: Faculty of Engineering

Abstract

This project will research the use of strategically-positioned, easy to wear, vibrating devices for improving engineers' understanding of 3D spaces in virtual environments. The system will use low-cost technologies and support day-to-day engineering design work.

Currently, the majority of engineering design is conducted on desktop systems. These do not allow engineers to experience the 3D spaces they are designing, thus limiting design success in applications with human users, such as component access routes for assembly and reach investigations for a manufacturing line. Engineering for users currently relies mainly on physical prototypes, which are expensive to produce and may not reflect current design intent due the time taken to build them. Digital human modelling, in which CAD representations of humans are used for ergonomics investigations, offer some benefits but are not suitable for complex motions and do not provide subjective responses. Design solutions can be viewed and experienced in virtual reality such as CAVEs, but depth perception in virtual environments can be inaccurate, leading to rejection of this technology by engineers, or unsound decision-making. Moreover, CAVEs are expensive and gaining access to them can be difficult for engineers conducting everyday design and analysis work.

This project will determine whether a small number of worn haptic (sense of touch) devices can improve spatial awareness in virtual environments. When viewing a 3D environment, a collision between the engineer's body and the virtual object will be indicated by vibrations on one of these devices. The devices may not be located exactly at the point of contact, for example, the engineer's elbow may contact the limits of the space they are designing, but the haptic cue may be experienced on the forearm. This approach will allow for a smaller number of devices, which will make the system more wearable than current haptic suits and more acceptable to engineers. No previous research has addressed such a challenge. The project outcome will be improvements in engineers' understanding of 3D spaces, which will increase the robustness with which decisions are made about designs during the early phases, which could reduce engineering development time and the use of expensive physical prototypes.

The project will adapt low-cost off-the-shelf technologies to retain focus on affordable solutions which are accessible to engineers. To avoid the inconvenience of marker-based body tracking suits, a markerless system will capture the user's movements and display their body within the virtual environment. Laboratory research will first determine the perceptual advantages afforded by the system by comparing spatial understanding with multi-sensory feedback to a vision-only condition. Furthermore, evaluation will be made of sensory illusions, in which the point of vibrotactile sensation differs to the point of contact between the user's body and the virtual object as seen in a stereoscopic display. Following on from this laboratory study, the system will be tested in design and manufacturing use cases by engineers. This will determine the benefits of multi-modal stimulation on spatial awareness, but will also evaluate the usability and acceptance of the system in an engineering environment.

Planned Impact

This research will benefit engineers who design products and workplaces for human users. The research results will inform a new technique for experiencing designs during the CAD phase, before physical prototypes are available. In this way they will be able to make higher-validity decisions about their designs, which will reduce the costs associated with physical prototypes. Using virtual testing will allow them to evaluate a larger number of iterations in a shorter space of time than physical parts testing, thus resulting in faster and better-informed design solutions. Engineers are likely to achieve satisfactory solutions in a shorter space of time, thus reducing time to market and further improving their organisation's competitiveness. Furthermore, an emphasis of this project is on the use of low-cost technologies. Engineering companies will benefit by avoiding the costs associated with high-end visualisation technologies such as CAVEs. The low-cost technologies will increase their ubiquity, rather than only providing VR systems in dedicated laboratories with restricted access and complex data set up processes. These technologies will also provide immersive design capabilities for engineers who cannot currently access a CAVE, for example SMEs or engineers in developing countries.
Engineers will be able to optimise the suitability of their designs for human users as part of a proactive approach, thus reducing the ergonomics injuries associated with poorly designed workstations, for example on manufacturing lines. Thus, human users of the engineering solutions will also benefit from the work.
The impact will be expected most immediately through adoption of the research findings into the development of virtual reality technology systems. This will be supported through the involvement of academic, industrial and technology developer partners, including Jaguar Land Rover, Holovis, ENSAM, Virginia Tech, Bauhaus-Universitat Weimar, and the Manufacturing Technology Centre, and dissemination activities at their sites.
 
Description In this project we have developed and tested a low-cost haptic feedback system for engineering design using off-the-shelf technologies. The key findings from our studies are as follows: Study 1: We tested early prototypes which included vibrotactile devices (tactors) to deliver haptic feedback. We compared a full haptic feedback system with our "sensory illusions" prototype, in which the haptic feedback was delivered near to but not exactly at the point at which the user's body collides with a virtual object, meaning that the user can wear fewer tactors. Our participants all had backgrounds in Engineering. We found that the sensory illusions prototype scored better on subjective measures such as acceptability for use in the workplace context. Participants saw the value in vibrotactile feedback, and the majority felt that the VR system was more appropriate overall for engineering design work with haptic feedback than without, with 47% choosing the sensory illusions system as most appropriate, 35% the full haptic system, and only 18% the system without haptic feedback. However, this study revealed issues with off-the-shelf technologies including inaccuracies in markerless motion tracking, and we were unable to assess with sufficient accuracy whether the haptic feedback would improve performance in engineering design tasks. Study 2: Following iteration of the prototypes, we further investigated the benefits of the sensory illusions haptic feedback system. We found that participants were better able to judge reach space with haptic feedback than without when they were specifically instructed to attend to it, but that if not instructed participants would tend to make inaccurate judgements based on the distorted visual perception through the low-cost HMD. We also found that the haptic feedback improved performance in a timed target-touching task requiring high accuracy, with participants making fewer errors when haptic feedback was enabled. Furthermore we found that when the participants could not see their avatar, spatial judgements were dramatically improved by haptic feedback. Study 3: An advanced reduced haptic prototype was taken to a UK automotive company, where we ran feedback sessions with design engineers specialising in ergonomics and human factors demonstrating the use case of vehicle cockpit design. Feedback was positive overall and participants felt that the system could be useful in early stages of design, reducing the need for physical prototypes in this phase of development. While the main benefits of a multimodal VR system found in these studies were expected based on prior research, the novel contribution of this research is in demonstrating these benefits (a) in a specific engineering design use case, and (b) using only low-cost, easily available technologies, which would likely be accessible to SMEs. The project has also generated a list of requirements for multimodal VR systems in engineering design, and provides guidance for the development of technologies in this context.
Exploitation Route The prototype has demonstrated value, but would require refinement to be used in industry. Further work to develop the prototype into a market-ready system would be highly beneficial and should involve collaboration between research institutions, engineering design partners, and hardware developers. It is proposed that follow-on funding for product development should be identified to achieve this. Our work also highlights the need for further exploratory research particularly around the following challenges: 1) further investigation and comparison of markerless and marker-based motion tracking systems and their practicality in the workplace; 2) investigation of the most effective configuration of vibrotactile feedback to support accurate spatial judgements in design tasks, such as the intensity required for different body parts and the duration and pattern of vibrations; 3) investigation of individual differences and expertise, as our results suggest that the less well participants perform in making spatial judgements in VR, the more they benefit from the addition of haptic feedback; 4) investigation into the use of the multimodal feedback system as a means of training engineers to make accurate spatial judgements; it is proposed based on qualitative feedback from this project that following use of the multimodal feedback system, engineers may be better able to make spatial judgements in VR even when the haptic feedback is subsequently removed. These further areas to build on the project findings are in addition to ongoing follow-on work involving ultrasonic haptic feedback.
Sectors Creative Economy

Digital/Communication/Information Technologies (including Software)

Manufacturing

including Industrial Biotechology

Transport

Other

 
Description JLR Cognitive and Metabolic Workload analysis
Amount £39,300 (GGP)
Organisation Jaguar Land Rover Automotive PLC 
Department Jaguar Land Rover
Sector Private
Country United Kingdom
Start 02/2020 
End 05/2020
 
Description JLR Global Manufacturing Innovation ESOW - Novel Training Technologies
Amount £44,890 (GBP)
Organisation Jaguar Land Rover Automotive PLC 
Department Jaguar Land Rover
Sector Private
Country United Kingdom
Start 02/2018 
End 07/2018
 
Description Research Development Fund
Amount £179,076 (GBP)
Organisation Institution of Occupational Safety and Health (IOSH) 
Sector Academic/University
Country United Kingdom
Start 02/2017 
End 06/2019
 
Description Ultrasound haptic feedback for multimodal interaction with virtual environments
Amount £97,852 (GBP)
Funding ID 1937993 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2017 
End 03/2022
 
Description Research collaboration with ultrahaptics 
Organisation Ultrahaptics
Country United Kingdom 
Sector Private 
PI Contribution A PhD project researching the use of Ultrahaptics technology in the context of engineering design, supervised by Dr Glyn Lawson. Research findings will be integrated into Ultrahaptics' portfolio of solutions.
Collaborator Contribution Use of Ultrahaptics evaluation kit; industrial supervisor for PhD student; conference cost sponsorship; hosting an internship (including associated costs), and potential dissemination opportunities through trade shows or similar events.
Impact PhD funded, co-supervised between the University of Nottingham and Ultrahaptics
Start Year 2017
 
Title A low-cost multimodal VR system for engineering design 
Description We have developed a system incorporating low-cost, easily available technologies to support engineering design. The system combines a head-mounted display and worn vibrotactile devices controlled by Arduino to provide a multimodal experience of the virtual environment. Haptic feedback delivered through the vibrotactile devices is activated when the user's body collides with a surface in the VE, improving spatial judgements. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2017 
Impact The system has been demonstrated to reduce errors in tasks requiring high levels of accuracy. 
 
Description Ada Lovelace Day at The University of Nottingham 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact 5 groups of approximately 20 secondary school children were shown around the Faculty of Engineering Ada Lovelace outreach event. This included a demonstration of the Sensory Illusions multimodal virtual environment. This was a popular demonstration among the school children, who responded positively.
Year(s) Of Engagement Activity 2016
URL https://www.nottingham.ac.uk/engineering/schoolsandcolleges/event/events/ada-lovelace-day-october-20...
 
Description Demonstration at Designing Interactive Systems (DIS 17) conference 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact The latest prototype VR system was demonstrated. Delegates were invited to try the system using HMD and vibrotactile devices attached to wrist straps, while the VE was additionally displayed on a large monitor to enable other delegates to watch while the system was in use. The ability to make spatial judgements with and without haptic feedback was demonstrated, along with the example scenario of automotive design, for which a car interior was simulated. DIS attracted approx. 350 delegates from multiple disciplines (of whom about 75% were from academia). Our demonstration stand was particularly well attended and there were interesting discussions about how similar technologies could be used in this and other application areas. We were also able to disseminate results from our user studies. A summary paper for multidisciplinary audiences was included in the conference proceedings.
Year(s) Of Engagement Activity 2017
 
Description Demonstration at Engineering Interactive Computing Systems (EICS 17) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Demonstration of Sensory Illusions system at the EICS conference. Intended purpose to disseminate the work to experts in Computer Science field. Associated paper published in the conference proceedings, detailing technical specifications and challenges.
Year(s) Of Engagement Activity 2017
 
Description Demonstration at ImmerseUK academia and industry sub-group meeting at Ravensbourne College 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact This was a meeting of the "Academia-Industry" sub-group of the ImmerseUK network. 40 attendees, approximately half each from industry and academia met to discuss the challenges of working together in the VR area. During this meeting, Dr Glyn Lawson and Dr Tess Roper made a demonstration of a technology prototype from the sensory illusions project.

We received particular interest from attendees from Jaguar Land Rover, Dundee University and Ravensbourne College.
Year(s) Of Engagement Activity 2017
 
Description Demonstration at the European Conference of Cognitive Ergonomics (ECCE 2016) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact The phase 1 technology from the Sensory Illusions project was demonstrated during the European Conference on Cognitive Ergonomics. The 51 conference delegates were mainly academics, but also included professionals and postgraduate students. Delegates were invited to view the technology which was demonstrated in all demo sessions.

The purpose was to raise awareness of the research in the human factors and ergonomics community. The technology received significant interest from delegates.
Year(s) Of Engagement Activity 2016
URL https://www.nottingham.ac.uk/conference/fac-eng/ecce16/index.aspx
 
Description Demonstration of the Sensory Illusions phase one technology to six visitors from Bentley 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact Six visitors from Bentley cars visited the University of Nottingham on 26th May 2016. These visitors were from the Ergonomics department, Vehicle Architecture, Instrument Panel and Console and Electrical departments. Each visitor was given a short presentation of the background to the Sensory Illusions project, and was invited to try the phase 1 technology set up. Afterwards, Bentley acknowledged that they'd seen research which they need to implement in their processes. We initiated discussions around a possible Knowledge Transfer Partnership.
Year(s) Of Engagement Activity 2016
 
Description Demonstration to Jaguar Landrover employees 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact Demonstrated the technology to a group of Human Factors experts at Jaguar Landrover. Objectives of the session were to disseminate the work, showing how it could contribute to JLR's design processes, as well as to collect feedback on the system and to continue existing industry/academia relationship. We received useful feedback from the session; participants saw the value of haptic feedback in VR, and gave suggestions for further improvements and future work. This feedback will be incorporated into relevant papers, and will be used to shape future research proposals.
Year(s) Of Engagement Activity 2017
 
Description Digitop seminar on Human Factors in Industry 4.0 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Public seminar on Human Factors in Industry 4.0. Raised awareness of the pros and cons of virtual reality in industrial applications.
Year(s) Of Engagement Activity 2020
 
Description Human Factors Research Group Showcase and Networking Event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Technologies from the project were demonstrated to alumni, businesses and post-graduate students at an open day.
Year(s) Of Engagement Activity 2017
URL http://www.nottingham.ac.uk/research/groups/human-factors-research-group/events/showcasing-event.asp...
 
Description Lecture at Thammasat University in Thailand 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Undergraduate students
Results and Impact Dr Glyn Lawson (PI) visited Thammasat University in March 2017 to give guest lectures on Automotive Ergonomics to the Undergraduate students on Mechanical Engineering BSc Programmes. He gave four lectures to different year groups over a week, reaching approximately 60 students. For each lecture, Dr Lawson showed a lightweight technology demonstrator of the Sensory Illusions project.
Year(s) Of Engagement Activity 2017
 
Description Lecture on Human Factors in Industry 4.0 on Digital Manufacturing 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Undergraduate students
Results and Impact Lecture to students on the Digital Manufacturing Level 4 module at the University of Nottingham, December 2022. Made specific reference to the Digitop toolkit, and to the Sensory Illusions research. Students demonstrated greater engagement with the subject area, as evidenced by their coursework submissions.
Year(s) Of Engagement Activity 2022
 
Description Lecture on Human Factors in Industry 4.0 on the Digital Manufacturing 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Undergraduate students
Results and Impact Lecture to students on the Digital Manufacturing Level 4 module at the University of Nottingham, December 2021. Made specific reference to the Digitop toolkit, and to the Sensory Illusions research. Students demonstrated greater engagement with the subject area, as evidenced by their coursework submissions.
Year(s) Of Engagement Activity 2021
 
Description Lecture on Human Factors in Industry 4.0 on the Digital Manufacturing 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Undergraduate students
Results and Impact Introduced Human Factors challenges in Digital Manufacturing to students on a Manufacturing UG MEng course, and made specific reference to the Digitop Toolkit & the Sensory Illusions research. Students demonstrated an increased understanding of human factors challenges in digital manufacturing as evidenced by their coursework submissions.
Year(s) Of Engagement Activity 2020
 
Description Network Plus: Industrial Systems in the Digital Age Conference 2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Poster presentation at the Network Plus: Industrial Systems in the Digital Age Conference 2017 - Looking Beyond Industry 4.0. This conference brought together colleagues from manufacturing, digital economy, ICT, design, Human Factors, ICT and business communities. The poster presentation showed the value of using VE in engineering design processes and we discussed the use of affordable haptic feedback based on the research from this project.
Year(s) Of Engagement Activity 2017
 
Description Presentation at Ultrahaptics, Bristol 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Presentation of research and study results along with a demonstration of the technology to Ultrahaptics, a UK company working with ultrasonic haptic feedback technology. Aim was to establish opportunities for future research collaborations as well as to show the potential benefits of haptic feedback in engineering design, which is a field which Ultrahaptics had not yet explored. The presentation generated interest and was followed by discussions of how we could extend our research by using ultrahaptics technology, including postgraduate research supervised by the University of Nottingham and supported by Ultrahaptics. A follow-up meeting in Nottingham was arranged for further demonstration and to develop the proposal for a PhD project.
Year(s) Of Engagement Activity 2017