Designing Virtual Humans for the Digital Future: A Geometric Deep Learning Perspective
Lead Research Organisation:
Imperial College London
Department Name: Computing
Abstract
Virtual embodiment aims at creating digital environments in which subjects identify with their avatars. The current virtual reality applications suffer from the uncanny valley effect where the unrealistic body and motion reconstruction causes an unsettling feeling of interacting with fake agents. We propose designing virtual humans aesthetically indistinguishable from real people that act in a realistic way. While the problem seems to be Turing test-complete, we approach this from a visual perspective where the agent's representation is learned from images, videos, and 3D data. We show that incorporating methods from the emerging field of Geometric Deep Learning allows us to create realistic body models and reconstruct them from images.
Planned Impact
The proposed CDT will have an impact on most industrial sectors and organisations whose strategic objectives are underpinned by High Performance Embedded and Distributed Systems. The UK has an acute shortage of professionals with the relevant skills and competence to cover cross-layer design, analysis, optimisation and verification of such systems, since current PhD research often involves an isolated, narrow area.
As an example, Dr. Khaled Benkrid, ARM's Worldwide University Programme Manager, says in his statement of support that: "The shortage of highly-skilled professionals with PhD qualification is a real problem for us. While we are still enjoying a commanding lead in supplying semiconductor intellectual property for the development of low power, high performance digital products, this lead cannot be sustained without the availability of highly-skilled professionals. The proposed centre would go a long way to address the skills problem."
There is growing adoption of heterogeneous systems, such as implantable biosensors communicating with cloud-based analysis and decision support services. Telecommunications, finance, transport, healthcare and government services are also increasingly incorporating complex functionality in computing systems while moving towards a mixture of pervasive and cloud computing to support key activities. There is a critical need for technical competencies and leadership skills that this CDT aims to provide. Many of our CDT research projects will be multidisciplinary, relating to applications in Healthcare Systems, Smart Cities, and the Information Society.
Our approach to maximising impact is to involve industry, alumni and external users in many aspects of the CDT: taking part in giving courses, providing internships and career advice, and contributing to student supervision, Masterclasses, and Advanced Study Institutes. In particular, every CDT student will undertake at least one internship with industry or a leading research institution during their studies. This will ensure foundational research is relevant to external users. The CDT students will be undertaking research projects relevant to the organisation sponsoring their studies, and to the wider industry. We anticipate that students will often be employed by these organisations on graduation, thereby facilitating the transfer of research to the sponsoring organisation. In addition, we will provide advice and support on entrepreneurship and funding to develop research into prototypes for subsequent commercialization where appropriate. Our Industrial and Alumni Liaison Officer will organise and manage these routes to impact.
The high-tech sector requires a breadth and depth of knowledge and training beyond the level of Masters, as illustrated by the large proportion of engineering and managerial staff with PhD degrees in leading companies in the US and continental Europe. We purposely design this CDT to produce future leaders in industry, academia, government, and SMEs. This CDT will be able to continue our success in training leaders for industry and research; examples of such Imperial graduates include:
Ian Foster, Director, Computation Institute, Argonne National Laboratory;
Nabeel Shirazi, Director, Xilinx;
Tyrone Grandison, Program Manager, IBM San Jose Research Lab;
Guido Jouret, General Manager and CTO, Emerging Technologies, Cisco;
Xeno Andriopoulos, Managing Director, Kinitron;
Constantine Goulimis, Founder and CEO, Greycon;
Alex Buckley, Java Specification Team leader, Oracle.
Both Departments of Computing and Electrical and Electronic Engineering have a strong record of start-up companies, such as CVIS, DNA Electronics, GeneOnyx, InforSense, IXICO, Nexeon and Toumaz. Students from this CDT will be able to tap into these start-up activities.
Imperial Innovations has an excellent track record of commercialising academic research. They will advise on the appropriate forms of licencing and investment.
As an example, Dr. Khaled Benkrid, ARM's Worldwide University Programme Manager, says in his statement of support that: "The shortage of highly-skilled professionals with PhD qualification is a real problem for us. While we are still enjoying a commanding lead in supplying semiconductor intellectual property for the development of low power, high performance digital products, this lead cannot be sustained without the availability of highly-skilled professionals. The proposed centre would go a long way to address the skills problem."
There is growing adoption of heterogeneous systems, such as implantable biosensors communicating with cloud-based analysis and decision support services. Telecommunications, finance, transport, healthcare and government services are also increasingly incorporating complex functionality in computing systems while moving towards a mixture of pervasive and cloud computing to support key activities. There is a critical need for technical competencies and leadership skills that this CDT aims to provide. Many of our CDT research projects will be multidisciplinary, relating to applications in Healthcare Systems, Smart Cities, and the Information Society.
Our approach to maximising impact is to involve industry, alumni and external users in many aspects of the CDT: taking part in giving courses, providing internships and career advice, and contributing to student supervision, Masterclasses, and Advanced Study Institutes. In particular, every CDT student will undertake at least one internship with industry or a leading research institution during their studies. This will ensure foundational research is relevant to external users. The CDT students will be undertaking research projects relevant to the organisation sponsoring their studies, and to the wider industry. We anticipate that students will often be employed by these organisations on graduation, thereby facilitating the transfer of research to the sponsoring organisation. In addition, we will provide advice and support on entrepreneurship and funding to develop research into prototypes for subsequent commercialization where appropriate. Our Industrial and Alumni Liaison Officer will organise and manage these routes to impact.
The high-tech sector requires a breadth and depth of knowledge and training beyond the level of Masters, as illustrated by the large proportion of engineering and managerial staff with PhD degrees in leading companies in the US and continental Europe. We purposely design this CDT to produce future leaders in industry, academia, government, and SMEs. This CDT will be able to continue our success in training leaders for industry and research; examples of such Imperial graduates include:
Ian Foster, Director, Computation Institute, Argonne National Laboratory;
Nabeel Shirazi, Director, Xilinx;
Tyrone Grandison, Program Manager, IBM San Jose Research Lab;
Guido Jouret, General Manager and CTO, Emerging Technologies, Cisco;
Xeno Andriopoulos, Managing Director, Kinitron;
Constantine Goulimis, Founder and CEO, Greycon;
Alex Buckley, Java Specification Team leader, Oracle.
Both Departments of Computing and Electrical and Electronic Engineering have a strong record of start-up companies, such as CVIS, DNA Electronics, GeneOnyx, InforSense, IXICO, Nexeon and Toumaz. Students from this CDT will be able to tap into these start-up activities.
Imperial Innovations has an excellent track record of commercialising academic research. They will advise on the appropriate forms of licencing and investment.
Organisations
| Description | We have advanced the field of hand reconstruction and modelling by addressing its major weaknesses: performance in non-laboratory environments, inference time, data availability, shape diversity and bone length disentanglement, and texture synthesis. We have showed that neural networks are powerful methods for representing highly-articulated non-linear hand deformations in a compact mesh-convolutional decoder while improving speed and accuracy. We have addressed the problem of collecting dense 3D hand mesh annotations for RGB images by building an automated system for parametric model fitting and we made the first in the wild dataset for 3D hand pose estimation and mesh reconstruction publicly available. Furthermore, we have proposed a novel weakly-supervised neural network for hand mesh reconstruction. The system obtains unprecedented performance on the in the wild scenario while working at 250 FPS on mobile devices. We have also collected the largest and most diverse dataset of 3D hand scans in Science Museum in London. Afterwards, we processed over half a million scans to a common representation. This allowed us to perform statistical analysis of shape and texture variations. Specifically, we have built a deformable model of the human hand that disentangles bone length variability from other identity-specific traits. We have showed that incorporating this model to our weakly-supervised hand reconstruction network improves its state-of-the-art performance even further. We also took advantage of our large-scale hand scans dataset to learn high-resolution skin texture variations which has not been previously possible. |
| Exploitation Route | We argue that robust systems for hand mesh reconstruction are crucial for development of augmented/mixed/virtual reality technology by enabling intuitive control and interaction with the digital objects. Such systems also find applications in motion tracking, learning from demonstration / behavioural cloning, virtual telepresence, sign language recognition, healthcare, realistic prosthetic design, and immersive gaming. The proposed hand reconstruction network and statistical deformable models have been successfully incorporated into commercial products offering novel augmented reality experiences. |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software) |
| Description | The proposed hand reconstruction networks and statistical deformable models have been successfully incorporated into commercial products offering novel augmented reality experiences. |
| First Year Of Impact | 2019 |
| Sector | Creative Economy,Digital/Communication/Information Technologies (including Software) |
| Impact Types | Cultural,Economic |