High Quality 3D Geometry and Appearance Reconstruction of Non-Rigidly Deforming Objects using Low-Cost RGB-D Cameras
Lead Research Organisation:
Cardiff University
Department Name: Computer Science
Abstract
Capturing and reconstruction of high quality 3D geometry and the appearance of non-rigidly deforming objects, such as the dynamics of human actions, is essential for many applications, including movie and game production in the creative industries, Virtual Reality (VR) videoconferencing, analysing human behaviour for healthcare monitoring and sports analysis etc. Despite great effort, it is still a challenging problem, especially when large-scale deformations are involved: self-occlusion, subtle geometry and appearance change (e.g. wrinkles of skin and clothing) all contribute to the difficulty.
This research aims to advance the state of the art by investigating novel data-driven techniques to address fundamental challenges. Low-cost RGB-Depth cameras have become more capable in recent years and will be used in the research to make the techniques widely useful. We will develop a new joint representation and analysis technique for both geometry and appearance, to effectively encode geometric and appearance change during non-rigid deformation. The plausible deformation and change in appearance typically form a low dimensional manifold embedded in this joint space. To address the issues of noise and incompleteness in the scanned data, machine learning techniques such as manifold learning will be exploited. This will effectively utilise information from any previous scans to fill the gaps and improve the quality of reconstruction. An optimisation framework will also be developed incorporating knowledge from the manifold as well as sparse priors.
To make the research feasible, the project is built on top of the supervisors' existing work on shape deformation (representation and shape space analysis), non-rigid registration, data-driven reconstruction and sparse models, all of which have recent publications in top journals. The project is related and complementary to two current Royal Society international collaborative projects with leading research institutions in China. The research and the PhD student will benefit greatly from these collaborations.
This research aims to advance the state of the art by investigating novel data-driven techniques to address fundamental challenges. Low-cost RGB-Depth cameras have become more capable in recent years and will be used in the research to make the techniques widely useful. We will develop a new joint representation and analysis technique for both geometry and appearance, to effectively encode geometric and appearance change during non-rigid deformation. The plausible deformation and change in appearance typically form a low dimensional manifold embedded in this joint space. To address the issues of noise and incompleteness in the scanned data, machine learning techniques such as manifold learning will be exploited. This will effectively utilise information from any previous scans to fill the gaps and improve the quality of reconstruction. An optimisation framework will also be developed incorporating knowledge from the manifold as well as sparse priors.
To make the research feasible, the project is built on top of the supervisors' existing work on shape deformation (representation and shape space analysis), non-rigid registration, data-driven reconstruction and sparse models, all of which have recent publications in top journals. The project is related and complementary to two current Royal Society international collaborative projects with leading research institutions in China. The research and the PhD student will benefit greatly from these collaborations.
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509449/1 | 01/10/2016 | 30/09/2021 | |||
| 1806028 | Studentship | EP/N509449/1 | 01/10/2016 | 31/03/2020 | Roberto Dyke |
| Description | We have investigated the challenges involved in understanding objects that my undergo stretching (e.g. Stretch Armstrong). We have developed an automatic method that is capable of measuring and describing this stretching independently of bending properties. We have transferred this approach to automatically find the relationship between different objects (e.g., two human men, or a man and a woman, etc.). Furthermore, we have identified that there is no current benchmark that is suitable for measuring the performance of methods for automatically finding relationships between objects in the face of different degrees of stretching and bending. Thus, we have produced our own dataset that addresses this issue. |
| Exploitation Route | Our work addresses a fundamental problem in our area that is under-researched. This has far-reaching applications across many fields that study or exploit 3D geometry, such as: archaeology (e.g. studying the development of bones), dentistry (e.g. studying changes in teeth), computer vision (e.g. studying action recognition), computer graphics (e.g. transferring facial animations), etc.). Our investigations prove our method improves upon previous works, thus others attempting to solve similar problems in the future may refer to our approach. |
| Sectors | Digital/Communication/Information Technologies (including Software),Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Manufacturing, including Industrial Biotechology,Pharmaceuticals and Medical Biotechnology,Retail,Other |
| Title | Non-Rigid Shapes with Isometric and Non-Isometric Deformations |
| Description | This is a collection of non-rigid shapes that exhibit various isometric and non-isometric deformations. The primary purpose of this dataset is to provide a benchmark to compare shape correspondence generation methods. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | This dataset is currently being used as a basis for a 3D Shape Retrieval Contest (SHREC) track that sets out to evaluate the performance of state-of-the-art methods. |
| URL | https://shrec19.cs.cf.ac.uk/ |
| Description | HDFD---A High Deformation Facial Dynamics Benchmark for Evaluation of Non-Rigid Surface Registration and Classification |
| Organisation | Swansea University |
| Department | Department of Computer Science |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | We contributed our expertise and assisted with data processing during this project. |
| Collaborator Contribution | Our partners conducted data collection, and subsequent data analysis of our results and report writing. |
| Impact | The collaboration resulted in the following article. Andrews, G., Endean, S., Dyke, R., Lai, Y., Ffrancon, G., & Tam, G. K. (2018). HDFD---A High Deformation Facial Dynamics Benchmark for Evaluation of Non-Rigid Surface Registration and Classification. arXiv preprint arXiv:1807.03354. |
| Start Year | 2018 |