Towards Zero Prototyping of Factory Layouts and Operations Using Novel Gaming and Immersive Technologies

Lead Research Organisation: Cranfield University
Department Name: School of Water, Energy and Environment

Abstract

Aim: The research aims to integrate Discrete Event Simulation (DES) with recent innovations in gaming interface technologies to incorporate novel immersive Virtual Reality (VR) and live event feedback capabilities into DES. The resulting technology will address the barriers that prevent more effective use of modelling and simulation in designing newbuilt factories and managing the operations of existing factories. The research results will be demonstrated by integrating Lanner's DES tool WITNESS with Oculus Rift, a VR goggle, and Microsoft Kinect, an integrated 3D sensing device, hence incorporating immersive VR (Immersive WITNESSS or IW) and live event feedback capabilities (Live WITNESS or LW) into WITNESS.
What is Immersive WITNESS (IW)? IW is the integration of the latest consumer device VR technology (Oculus Rift) with Lanner's DES tool WITNESS. At the moment, WITNESS mainly relies on standard (non-immersive) display technologies.
VR was a very popular topic in the 1990s, but interest waned away rapidly due to a gap between expectations and technological limitations. Recent developments have closed the gap considerably, opening up an opportunity for a fresh attempt to achieve the full potential of VR in manufacturing simulations.
How does IW bring value? The coupling of DES and VR opens up a new dimension for process simulation. Among the benefits cited in literature are increased involvement from decision makers resulting in an improved model and better understanding of the processes, increased confidence in the model, quicker and easier detection of errors and evaluation of model behaviour, facilitation of the generation of ideas, and better communication with clients and other stakeholders.
Oculus provides unprecedented immersion in a VR world at a fraction of the cost of traditional devices. Furthermore, new developments in human-computer interfaces (Leap Motion, Sixense STEM, Myo, Kinect) have opened up potentially useful
ways for users to interact with the virtual world.
What is Live WITNESS (LW)? LW is a novel means to synchronise DES simulation with real-world through the use of an integrated 3D-sensing consumer device (Microsoft Kinect) that can provide a cheap, robust and unobtrusive sensor system on the shop-floor to view a given scene, observe a process and generate appropriate signals to update the DES model.
How does LW bring value? The drawback of using DES is the effort and cost of processing the input data from various data sources to ensure valid simulation results. Attempts have been made for gathering data from existing ICT systems; dealing
with legacy data remains a great challenge. LW side-steps this problem by using a cheaply generated data stream. Success will mean that DES models can accurately represent real-world processes at all times.
Why are IW and LW being proposed together to create Immersive/Live WITNESS (I/LW)? 3D sensing and VR are complementary technologies; the former helps in constructing an accurate virtual mirror of the real world and the latter
visualises the resulting virtual world. For example, Kinect provides a way to rapidly generate virtual world that mirrors reality, which can be viewed and edited using Oculus. This project provides a setting where both technologies can complement each other. The proposed approach of developing IW and LW at the same time will also reduce risks whilst creating more value than if each component were developed separately. Cranfield University will lead the integrated development of I/LW.
Choice of technologies: There have been many recent developments in human-computer interface technologies rooted in
consumer devices, such as Oculus Rift and Microsoft Kinect. In manufacturing, these technologies have the potential to deliver an unprecedented level of immersiveness, detail and real-time accuracy to DES such that the virtual factory always
accurately mirrors the real factory. This is a new project with no direct competitors.

Planned Impact

Economic benefits for the consortium: The main beneficiaries of this project are the consortium members. The research
aims to integrate Discrete Event Simulation (DES) with recent innovations in gaming interface technologies to incorporate
novel immersive Virtual Reality (VR) and live event feedback capabilities into DES. The resulting technology will address
the barriers that prevent more effective use of modelling and simulation in designing new-built factories and managing the
operations of existing factories. The research results will be demonstrated by integrating Lanner's DES tool WITNESS with
Oculus Rift, a VR goggle, and Microsoft Kinect, an integrated 3D sensing device, hence incorporating immersive VR
(Immersive WITNESSS or IW) and live event feedback capabilities (Live WITNESS or LW) into WITNESS. The main
deliverables (IW, LW and integrated I/LW) will be commercialised by the consortium as add-on modules to WITNESS.
These deliverables will create direct economic benefits for all consortium partners through a revenue sharing agreement.
Three exploitation paths are envisioned: (i) The integrated I/LW will initially be licensed to large manufacturers as a standard package catering to the design and operations of factories. (ii) Each component (IW and LW) can also be licensed
to customers as standard packages, and as customised packages for specialised needs. (iii) Exploitation may also happen
through Alstom, which could adopt the deliverables within the company, towards deployment as a business wide system in
addition to the design of factory layouts.
Economic benefits for the UK and the EEA: This project will benefit the manufacturing and ICT companies in the UK,
particularly the manufacturers who use modelling and simulation in designing new-built factories and managing the
operations of existing factories, "manufacturers' manufacturers" who design and implement manufacturing systems,
designers and developers of machine tools and industrial robots, and ICT providers involved in the simulation and
optimisation of manufacturing systems, gaming technologies and immersive virtual environments. For manufacturing
companies, this project has the potential to deliver an unprecedented level of immersiveness, detail and real-time accuracy
to DES simulation such that the virtual factory always accurately mirrors the real factory. This will allow an engineering
team to reconstruct a shop-floor into a VR world, then walk through the 3D model that is updated live from the shop-floor,
make exploratory changes in the VR world directly, and verify and validate outcomes in VR. This project will provide the
ICT companies with a novel approach to integrate DES tools with state-of-the-art VR and 3D sensing technologies. The
EEA will benefit because Alstom and other manufacturing companies will remain competitive and continue to support their
partners/vendors in the EEA. The proposed technology will be useful for any manufacturing company that uses modelling
and simulation in designing new-built factories and managing the operations of existing factories, such as energy
equipment, train, automotive and aerospace manufacturers.
Environmental and social benefits: This project will have environmental and social benefits through right-first-time building
of factories and through more efficient factories in terms of energy and material usage. Reducing factory test runs and the
time needed for production ramp-up will reduce consumption of resources. By ensuring that factory layout modelling
incorporates live data about human movement, there will also be improvements in health and safety and workers' quality of life.

Publications

10 25 50
 
Title Video of final demonstrator 
Description This video shows the end-to-end system demonstrator of project results. 
Type Of Art Film/Video/Animation 
Year Produced 2016 
Impact This video has been shown to potential partners and collaborators with good feedback, resulting in multiple follow-on project opportunities. 
URL https://www.youtube.com/watch?v=8U6u9N1nsqM
 
Description We have discovered that it is possible and valuable to develop convincing factory environments through the use of consumer-grade Virtual Reality. In this virtual environment, we can experiment with various factory layout possibilities and see which ones work best for a given scenario. As the virtual environment is linked with manufacturing simulation, we can also "run" the factory through various scenarios. For example, we can stop individual machines to see how this affects overall production. We can also experiment with various ways of addressing shop floor issues, thereby reducing the risk in actual implementation. Through simulation, we can discover layout and operational issues before any money is spent on building the actual factory. This translates into cost and time effectiveness.

We have also discovered that it is possible and valuable to feed data back into the manufacturing process simulation from the actual shop floor using consumer-grade sensors. One key contribution within this is our discovery of a way to train a computer equipped with a camera to recognise what it is looking at on a manufacturing floor. Training was achieved through the use of pre-defined simulation models and a database of shop floor CAD geometries. This enables the computer to infer manual handling tasks as well as the status of machines on the shop floor. From this, performance data of the actual process can be recorded. Comparison of this data with simulated data enables us to highlight potential bottlenecks and production shortfalls.

Finally, by combining the virtual factory environment with live data feedback, we have developed a way to create a "digital twin" that can be used by factory managers to increase productivity while significantly de-risking the impact of factory layout changes.
Exploitation Route Connecting Virtual Reality (VR) and Process Simulation (PS) in a hybrid framework is a powerful combination that is currently under-researched. We have shown that VR gives users the power to experience a factory before it is constructed while PS enables users to conduct predictive simulations to analyse the effects of various decisions on the factory. Other researchers can take forward our framework by combining VR with different types of PS, such as Agent-Based Simulation or System Dynamics Simulation. Our research on bootstrapping vision systems using process models can have great value in many industrial applications, for example, it can be used to upgrade the connectivity of legacy machines in a factory. In addition, our work on predictive simulation fed by real-time shop floor data can be expanded further by utilising the latest advances in machine learning techniques, e.g. deep learning. Finally, these findings could be applied to a range of industrial sectors, such as power, automotive, aerospace, construction and healthcare management.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Energy,Manufacturing, including Industrial Biotechology

 
Description The project uses low-cost new innovative technologies within the gaming sector in the form of virtual reality and human-computer interface sensory technologies. Developing Machine Learning Techniques to Mine Sensor Data: This project started with the recognition of the potential of sensor data feeds for better decisions in smart manufacturing. Towards this, mining of sensor data feeds is achieved using machine learning techniques. These techniques utilise previously generated simulation models and CAD models to train computer vision algorithms. The trained computer vision algorithms are capable of: (1) inferring the status of machines as well as material flows in a factory, (2) inferring the status of manual assembly tasks, and (3) updating simulation models. The updated models enable better prediction of future manufacturing states, for example, whether a manufacturing target will be achieved at the current production rate. Going forward, we envisage that these project outputs will be taken up by companies and fed into their smart manufacturing initiatives. Exploiting Virtual Reality Technologies for Zero Prototyping of Factory Layouts: We have developed techniques that enable users to simulate material and people flows on a shop floor. We can update the status of machines in the virtual reality environment and see the effect of the changes on production rates and targets. This helps decision makers experiment with various factory configurations without the need to disrupt a manufacturing line. We envisage that these outputs will be eventually assimilated into simulation products in the near future. The users of these products will have an option to use virtual reality immersion to show all potential stakeholders how a future planned factory will operate. These outputs have enabled the project partner, Lanner Group, to showcase futuristic capabilities that are within reach at conferences and user events. The tangible demonstrators developed by the project have enabled the project partner, GE Power, to connect with other teams across the wider GE organisation and inform a number of future applications within the business. Societal Impact: The project team was also active in engaging with other audiences. On 22 March 2016, students from Holywell Middle School visited the Manufacturing Informatics Centre at Cranfield University. Eighteen students accompanied by three staff members were met by Professor Ashutosh Tiwari and his team. Selected project outputs, such as the use of Microsoft Kinect sensors on manufacturing shop floors to help factories improve their productivity, were showcased to the students. We believe that this public engagement exercise has enabled the team to inspire the next generation of students to take up manufacturing. Future: This project will have environmental and social benefits through right-first-time building of factories and through more efficient factories in terms of energy and material usage. Reducing factory test runs and the time needed for production ramp-up will reduce consumption of resources. For example, a research project with the Manufacturing Technology Centre looked at how Virtual Reality could be used to develop digital twins of a physical robot cell towards exploring human-robot collaboration strategies. The hypothesis is that the lessons learnt in a virtual environment will inform policies on safe human-robot collaboration.
First Year Of Impact 2015
Sector Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Energy,Manufacturing, including Industrial Biotechology
Impact Types Societal,Economic

 
Description EPSRC High Value Manufacturing Catapult Fellowship
Amount £67,608 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 12/2015 
End 11/2018
 
Description Industrial Systems in the Digital Age
Amount £59,623 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 06/2017 
End 05/2018
 
Description Innovate UK - Industrial Strategy Challenge Fund
Amount £670,360 (GBP)
Funding ID 103554 
Organisation Innovate UK 
Sector Public
Country United Kingdom
Start 09/2017 
End 03/2019
 
Description Investigating human-robot collaboration strategies in a virtual reality environment (MSc group project)
Amount £12,000 (GBP)
Organisation Manufacturing Technology Centre (MTC) 
Sector Private
Country United Kingdom
Start 02/2017 
End 05/2017
 
Title Updating Factory Layout Models Based on Computer Vision 
Description This method uses a camera system to read and infer the state of a manufacturing shop floor. Based on this, our algorithms update a virtual model of the factory. The goal is to achieve accurate real time predictive analytics of factory shop floor. 
Type Of Material Improvements to research infrastructure 
Year Produced 2016 
Provided To Others? Yes  
Impact This research is still ongoing but we foresee an industrial impact that would enable decision makers to have more accurate models of factories. 
 
Title Virtual Reality work flow for developing manufacturing digital twins 
Description We have developed a workflow that uses a set of software tools that use CAD models and shop floor measurements to convert physical spaces into virtual spaces. 
Type Of Material Improvements to research infrastructure 
Year Produced 2017 
Provided To Others? Yes  
Impact The approach is currently being investigated to keep track of maintenance updates in manufacturing plants. 
 
Title Virtual reality tool to investigate human-robot collaboration strategies 
Description We developed a virtual reality tool that provides a foundation to investigate human-industrial robot collaboration strategies in a safe environment. The hypothesis is that strategies developed in a virtual environment will inform kinematics control of real world industrial robots. 
Type Of Material Improvements to research infrastructure 
Year Produced 2017 
Provided To Others? Yes  
Impact We are currently working with MTC to develop a digital twin of an industrial robot cell. The aim is to measure human reaction times and behaviours in a safe way to inform policies on human-robot collaboration. 
 
Description Automated conversion of physical spaces into virtual spaces for updating maintenance records 
Organisation General Electric
Country United States 
Sector Private 
PI Contribution Cranfield Team will be enhancing the virtual reality software tools developed as part of this project. Due to the limitations of the Microsoft Kinect, an industrial laser scanner will be used instead. Lessons learnt in manually converting physical spaces to virtual spaces will be exploited to develop an automated tool to achieve this.
Collaborator Contribution General Electric will be providing the use case as well as the industrial laser scanner.
Impact This will result in a part-funded project with Cranfield University.
Start Year 2017
 
Description Collaboration with Alstom Transport to investigate the potential use of low cost robotic solutions in train maintenance 
Organisation Alstom
Department Alstom UK
Country United Kingdom 
Sector Private 
PI Contribution Providing technical information regarding potential use cases. Taking the lead in writing an innovate UK proposal.
Collaborator Contribution Providing market need for low cost robotic train maintenance as well as use cases for feasibility studies
Impact Resulted in submission of an innovate UK proposal.
Start Year 2015
 
Description Creating a digital twin to investigate human-robot collaboration strategies in a virtual environment 
Organisation Manufacturing Technology Centre (MTC)
Country United Kingdom 
Sector Private 
PI Contribution Cranfield University is developing a digital twin of an actual physical robot cell at the Manufacturing Technology Centre (MTC). We are also developing mechanisms that measure human reactions to robot movements. So far, we have seen that different people have different reaction times as well as responses. This will be used to inform collaborative strategies with the virtual robot. The collaborative strategies will then be uploaded onto the physical robot for real world use. This approach enables the consortium to investigate human-robot collaborative strategies in a safe manner and use the outputs to inform future government policies.
Collaborator Contribution The Manufacturing Technology Centre is providing the use case and the physical robot cell layout for tests.
Impact This has resulted in a paper submitted to the EEE/RSJ International Conference on Intelligent Robots and Systems.
Start Year 2017
 
Description Investigating the use of smart objects and robots for intelligent manufacturing 
Organisation General Electric
Country United States 
Sector Private 
PI Contribution The Cranfield Team are working with Global Robots and General Electric (GE) Garages to develop an innovate UK proposal to investigate the possibility of "smartening up" objects on the manufacturing line. The aim is to achieve flexible manufacturing through distributed computing to reduce the computational burden on robotic arms used on manufacturing lines.
Collaborator Contribution Global Robots will be providing refurbished robots for demonstrators while GE Garages will investigate how the demonstrator could be used to develop their new line of products.
Impact One Innovate UK proposal with another being prepared
Start Year 2016
 
Description Investigating the use of smart objects and robots for intelligent manufacturing 
Organisation Global Robots
Country United Kingdom 
Sector Private 
PI Contribution The Cranfield Team are working with Global Robots and General Electric (GE) Garages to develop an innovate UK proposal to investigate the possibility of "smartening up" objects on the manufacturing line. The aim is to achieve flexible manufacturing through distributed computing to reduce the computational burden on robotic arms used on manufacturing lines.
Collaborator Contribution Global Robots will be providing refurbished robots for demonstrators while GE Garages will investigate how the demonstrator could be used to develop their new line of products.
Impact One Innovate UK proposal with another being prepared
Start Year 2016
 
Description Article in GE Reports 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact A GEReports.com article was published online to promote the project. The article outlines the work which has been developed throughout the project focussing on the link between Virtual Reality, visualisation and factory simulation models. The website has a global audience and hundreds of thousands of subscribers, including many journalists. A link to the piece has been included in subsequent GE Reports articles and in several internal news emails.
Year(s) Of Engagement Activity 2016
URL http://www.gereports.com/call-of-duty-this-woman-is-using-video-games-kinect-and-vr-to-make-actual-f...
 
Description Digital Technologies for Manufacturing Innovation: Embracing Industry 4.0 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact The research team was invited to this event to present its demonstrators on live simulation and immersive visualisation of factory environments. This enabled us to see the impact that our research will have on industry as well as identify how best to package the solutions being developed by the research team. The event was organised by the Institute for Advanced Manufacturing (ifAM) at the University of Nottingham, in collaboration with EPSRC and Innovate UK.
Year(s) Of Engagement Activity 2015
URL https://www.nottingham.ac.uk/conference/fac-eng/dtmi2015/index.aspx
 
Description GE Power Annual Science and Technology Day (Inaugural Event) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Around 80 GE engineers attended a one-day event to promote Digital activities inside the company so VR, Robotics, Optimisation & Automation were all featured. The company reported many positive comments from engineers who found the experience very interesting and informative. Having local universities and R&D organisations there brought an additional dimension to the event. They are confident that these initiatives will improve their links and knowledge transfer.
Year(s) Of Engagement Activity 2017
 
Description GE Power Leaders Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact In January 2016, a 'GE Power Leaders Workshop' was held in Hammersmith, London. As part of the workshop, top executives from different GE businesses were invited to attend various lectures and discussions. The project team presented the outputs from this research. Subsequently, the team was awarded the Best Showcase Award. The event was also an opportunity to discuss the project with key stakeholders within GE.
Year(s) Of Engagement Activity 2016
 
Description GE Science & Technology Day 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact On 21 February 2017, GE organised this event to showcase its research projects to interested parties within various GE businesses.
Year(s) Of Engagement Activity 2017
 
Description International industry-sponsored student engagement (China) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact A joint presentation between Cranfield team, Lanner Group (simulation software company), and Virtalis (virtual reality company) to convey project goals and benefits to industry-sponsored MSc students from COMAC aerospace company, China. As a result, there is now increased interest among the students to make use of the project's offerings, firstly in their individual projects, and finally in their company.
Year(s) Of Engagement Activity 2015
 
Description Middle School visit 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact On 22 March 2016, students from Holywell Middle School visited the Manufacturing Informatics Centre at Cranfield University. Eighteen students accompanied by three staff members were met by Professor Ashutosh Tiwari and his team. Researchers John Oyekan and Chika Mgbemena led a session on the use of Microsoft Kinect sensors on manufacturing shop floors to help factories improve their productivity. Windo Hutabarat and Boyang Song guided the visitors on the latest in immersive and augmented reality technologies such as Oculus Rift and Google Glass. The day concluded with a pub quiz style session led by Michael Farnsworth and Divya Tiwari, where every student was presented with a Cranfield souvenir kindly donated by the Vice-Chancellor.

The event was very successful, with positive comments from the students and their teachers. Some students have even asked at what age they can enter Cranfield University!
Year(s) Of Engagement Activity 2016
 
Description Quarterly Progress Meetings with Industrial Partners (GE and Lanner Group) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact This is a regular quarterly event to review the progress of the research collaboration between Cranfield, GE and Lanner Group. This event discusses the risks and mitigation strategies required to bring the project into successful conclusion. It informs the next set of technological products being developed and used by the industrial partners. This informs our dissemination and engagement plans including the choice of use cases.
Year(s) Of Engagement Activity 2015,2016,2017
 
Description UK delegation to India 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact The project team was part of a UK delegation to India in November 2016. This visit was used by the project team to disseminate the successful project outcomes. A summary of the visit is provided as follows:
1) The delegation visited Delhi as part of TECH Summit (7-9 Nov), Pune (9-10 Nov) and Bangalore (UK-India Advanced Manufacturing Innovation workshop, 11 Nov).
2) The project team prepared exhibition stands to demonstrate the potential of using simulation-driven Kinect and virtual reality on the shop floor to improve manufacturing processes.
3) An excellent response was received from stand visitors including VIP guests.
Year(s) Of Engagement Activity 2016
URL https://www.gov.uk/government/world-location-news/india-uk-tech-summit-2016
 
Description Workshop and Virtual Reality Demonstration 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact On 25 January 2016, a workshop was held at GE, representing 30 stakeholders from a large range of business sectors. The workshop included a presentation on the capabilities developed by the project team and the results of the project to date. The project team demonstrated videos of a maintenance workshop model as well as a desktop factory with a Microsoft Kinect picking up material movement times. The workshop was very well received with many encouraging comments, suggestions and follow up emails.
Year(s) Of Engagement Activity 2016