Multi-view computational scene modelling
Lead Research Organisation:
University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP
Abstract
In recent years, we have witnessed dramatic improvements in 3D reconstruction accuracy. With the advent of low cost depth sensors and more robust multi-view reconstruction algorithms, scene modelling techniques have become popularised and no longer limited to lab environments. However, despite their flexibility, existing techniques remain limited to scenes with simple surface reflectance properties (mostly diffuse). While more complex photometric approaches have been proposed to deal with such scenes, they tend to require expensive specialist equipment that cannot be deployed outside of the lab.
This project will fill this gap by developing more general and versatile digitisation techniques that can be applied to natural scenes with complex reflectance captured using consumer hardware. Building on recent advances in computer vision and computational photography, the project will introduce novel acquisition hardware and algorithms to infer depth together with other important attributes such as surface normal and BRDF. To further push the limits of modelling accuracy, the research will also consider complex physical phenomena which degrade reconstruction accuracy, such as sub-surface scattering and inter-reflections, and will explore how algorithms can be made more resilient to these effects.
The application of this research will be demonstrated in the context of the creative industries such as film, gaming and cultural heritage which all require the ability to create photo-realistic digital copies of assets. A particular emphasis will be placed on modelling natural or man-made scenes which have a complex surface reflectance (e.g. glossy materials). It is anticipated that this technology will find applications in many other areas of industry which require accurate modelling such as in robotics or the manufacturing industry.
This project will fill this gap by developing more general and versatile digitisation techniques that can be applied to natural scenes with complex reflectance captured using consumer hardware. Building on recent advances in computer vision and computational photography, the project will introduce novel acquisition hardware and algorithms to infer depth together with other important attributes such as surface normal and BRDF. To further push the limits of modelling accuracy, the research will also consider complex physical phenomena which degrade reconstruction accuracy, such as sub-surface scattering and inter-reflections, and will explore how algorithms can be made more resilient to these effects.
The application of this research will be demonstrated in the context of the creative industries such as film, gaming and cultural heritage which all require the ability to create photo-realistic digital copies of assets. A particular emphasis will be placed on modelling natural or man-made scenes which have a complex surface reflectance (e.g. glossy materials). It is anticipated that this technology will find applications in many other areas of industry which require accurate modelling such as in robotics or the manufacturing industry.
Organisations
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509772/1 | 01/10/2016 | 30/09/2021 | |||
| 1815219 | Studentship | EP/N509772/1 | 01/10/2016 | 31/07/2020 | Gianmarco Addari |
| Description | I have developed new tools for the digitisation of scenes with complex materials. More specifically, the research has since resulted in the following achievements: - Developed two novel methods based on a volumetric framework to perform full 3D reconstruction of real objects with complex surfaces, allowing for sub-millimetre accuracy reconstruction. This addresses an important limitation of existing digitisation techniques which tend to be limited to specific classes of objects (e.g. diffuse or sufficiently textured). These findings were disseminated as an oral presentation at the International Conference on Computer Vision Theory and Applications (VISAPP 2019). An extension of the first method presented was further developed which significantly improved the reconstruction results. - Produced a mesh based novel methodology to perform full 3D reconstruction of objects characterised by complex surfaces. The use of a mesh-based approach allowed for a further improvement in reconstruction precision with respect to the volumetric approaches. In addition to this method, a new approach to perform camera visibility estimation on such objects was proposed. These contributions were disseminated as a full paper at the 2019 European Conference on Visual Media Production (CVMP '19) through an oral presentation. - Built a prototype acquisition system using off-the-shelf components. This provides a low-cost solution to digitise a 3D scene from consumer hardware without the traditional limitations imposed by the scene material properties. The prototype was used to capture a new dataset which will be useful in the academic environment for future research on Helmholtz Stereopsis. - Systematically investigated possible configurations where reconstruction cannot be properly carried out by the approaches previously presented. In particular, surface position and surface normal ambiguities were analysed with respect to the cameras' positions relative to the reconstructed object. The findings were then tested on real scenes to measure the effect of the degenerate configurations on the 3D reconstruction and to serve as a quality score in the case of multiple hypotheses for a point's position. |
| Exploitation Route | The findings can be used in both academic and industrial settings: - In industry, the proposed methods and acquisition system could be used to produce assets for creative applications such as digital cinema, gaming, cultural heritage and design. The ability to perform non-contact digitisation of scenes with complex materials may also find applications in manufacturing (e.g. quality control, automation of processes using robotics). - In academia, the proposed methodology will most directly benefit researchers in the field of 3D computer vision. The proposed approach is likely to inform the development of other forms of reconstruction algorithms which build on the concept of Helmholtz reciprocity, including applications to dynamic scenes such as markerless motion capture. This may also benefit researchers working on reflectance estimation by providing a solution to a complementary problem (geometry estimation). |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software),Manufacturing, including Industrial Biotechology,Culture, Heritage, Museums and Collections |