Shape and Reflectance Acquisition of Complex Dynamic Scenes
Lead Research Organisation:
University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP
Abstract
Scene modelling is central to many applications in our society including quality control in manufacturing, robotics, medical imaging, visual effects production, cultural heritage and computer games. It requires accurate estimation of the scene's shape (its 3D surface geometry) and reflectance (how its surface reflects light). However, there is currently no method capable of capturing the shape and reflectance of dynamic scenes with complex surface reflectance (e.g. glossy surfaces). This lack of generic methods is problematic as it limits the applicability of existing techniques to scene categories which are not representative of the complexity of natural scenes and materials. This project will introduce a general framework to enable the capture of shape and reflectance of complex dynamic scenes thereby addressing an important gap in the field.
Current image or video-based shape estimation techniques rely on the assumption that the scene's surface reflectance is diffuse (it reflects light uniformly in all directions) or assume it is known a priori thus limiting the applicatibility to simple scenes. Reflectance estimation requires estimation of a 6-dimensional function (the BRDF) which describes how light is reflected at each surface point as a function of incident light direction and viewpoint direction. Due to high dimensionality, reflectance estimation remains limited to static scenes or requires use of expensive specialist equipment. At present, there is no method capable of accurately capturing both shape and reflectance of general dynamic scenes, yet scenes with complex unknown reflectance properties are omnipresent in our daily lives.
The proposed research will address this gap by introducing a novel framework which enables estimation of shape and reflectance for arbitrary dynamic scenes. The approach is based on two key scientific advances which tackle the high dimensionality issue of shape and reflectance estimation. First, a general methodology for decoupling shape estimation from reflectance estimation will be proposed; this will allow decomposition of the original high dimensional problem, which is ill-posed, into smaller sub-problems that are tractable. Second, a space-time formulation of reflectance estimation will be introduced; this will utilise dense surface tracking techniques to extend reflectance estimation to the temporal domain and thereby increase the number of observations available to overcome the inherently low number of observations at a single time instant. This will build on the PI's pioneering research in 3D reconstruction of scenes with arbitrary unknown reflectance properties and his expertise in dynamic scene reconstruction, surface tracking/animation and reflectance estimation.
This research represents a radical shift in scene modelling which will result in several major technical contributions: 1) a reflectance independent shape estimation methodology for dynamic scenes, 2) a non-rigid surface tracking method suitable for general scenes with complex and unknown reflectance and 3) a general and scalable reflectance estimation method for dynamic scenes. This will benefit all areas requiring accurate acquisition of shape and reflectance for real-world scenes with complex dynamic shape and reflectance without the requirement for complex and restrictive hardware setups; such scenes are a common occurrence in natural environments, manufacturing (metallic surfaces) and medical imaging (human tissue) but accurate capture of shape is not possible with existing approaches which assume diffuse reflectance and fail dramatically for such cases. This will achieve for the first time accurate modelling of dynamic scenes with arbitrary surface reflectance properties thus opening up novel avenues in scene modelling. The application of this technology will be demonstrated in digital cinema in collaboration with industrial partners to support the development of the next generation of visual effects.
Current image or video-based shape estimation techniques rely on the assumption that the scene's surface reflectance is diffuse (it reflects light uniformly in all directions) or assume it is known a priori thus limiting the applicatibility to simple scenes. Reflectance estimation requires estimation of a 6-dimensional function (the BRDF) which describes how light is reflected at each surface point as a function of incident light direction and viewpoint direction. Due to high dimensionality, reflectance estimation remains limited to static scenes or requires use of expensive specialist equipment. At present, there is no method capable of accurately capturing both shape and reflectance of general dynamic scenes, yet scenes with complex unknown reflectance properties are omnipresent in our daily lives.
The proposed research will address this gap by introducing a novel framework which enables estimation of shape and reflectance for arbitrary dynamic scenes. The approach is based on two key scientific advances which tackle the high dimensionality issue of shape and reflectance estimation. First, a general methodology for decoupling shape estimation from reflectance estimation will be proposed; this will allow decomposition of the original high dimensional problem, which is ill-posed, into smaller sub-problems that are tractable. Second, a space-time formulation of reflectance estimation will be introduced; this will utilise dense surface tracking techniques to extend reflectance estimation to the temporal domain and thereby increase the number of observations available to overcome the inherently low number of observations at a single time instant. This will build on the PI's pioneering research in 3D reconstruction of scenes with arbitrary unknown reflectance properties and his expertise in dynamic scene reconstruction, surface tracking/animation and reflectance estimation.
This research represents a radical shift in scene modelling which will result in several major technical contributions: 1) a reflectance independent shape estimation methodology for dynamic scenes, 2) a non-rigid surface tracking method suitable for general scenes with complex and unknown reflectance and 3) a general and scalable reflectance estimation method for dynamic scenes. This will benefit all areas requiring accurate acquisition of shape and reflectance for real-world scenes with complex dynamic shape and reflectance without the requirement for complex and restrictive hardware setups; such scenes are a common occurrence in natural environments, manufacturing (metallic surfaces) and medical imaging (human tissue) but accurate capture of shape is not possible with existing approaches which assume diffuse reflectance and fail dramatically for such cases. This will achieve for the first time accurate modelling of dynamic scenes with arbitrary surface reflectance properties thus opening up novel avenues in scene modelling. The application of this technology will be demonstrated in digital cinema in collaboration with industrial partners to support the development of the next generation of visual effects.
Planned Impact
The proposed research will have significant impact in the creative industry, particularly the visual media production community (film, television, game) where accurate shape and reflectance estimation are critical to produce photorealistic visual effects or digital characters as well as other communities where shape and reflectance estimation of scenes with complex surface properties are beneficial (cultural heritage, retail, manufacturing, robotics and medicine).
The creative industry is a rapidly evolving sector with a constant demand for novel technologies to improve the quality of content and reduce production times and costs for competitiveness. Currently the lack of suitable techniques for capture of scenes with complex surface reflectance properties is problematic as it places a tremendous burden on CG artists who need to manually generate photorealistic models. This increases post production times and costs and prevents novel applications due to budget constraints (e.g. limited photorealism in games or online fashion retail). This proposal is focused on achieving the fundamental scientific advances required to enable practical technologies for capture of dynamic shape with arbitrary surface reflectance. This will enable the generation of more sophisticated visual effects by removing current limitations on captured scenes with potential to significantly cut down production times and costs via reduction of the amount of manual interaction needed. For example, using current technology it is not possible to capture the performance of a fashion model or actress wearing a garment with complex reflectance properties such as silk to create a digital double. The project will combine CVSSP's expertise in computer vision and reconstruction with the expertise of world-leading companies in film VFX production (Double Negative) and film post-production tools (The Foundry). This will provide the platform to both deploy the technology for evaluation in production and enable technology transfer within the leading post-production tools (Nuke) for evaluation by production professionals. This will allow early prototyping and trialling of the technology to identify commercialisation opportunities.
The project will have a wide impact beyond the creative industries including the following communities (see further details in Pathways to Impact):
- Cultural heritage: the research will result in photorealistic digitisation techniques allowing preservation of cultural assets via creation of digital copies and supporting cultural enrichment of society by facilitating dissemination and access to cultural assets via the internet. Cultural assets considered will include artefacts held in museums and dynamic content such as traditional dances or performances.
- Online retail: the technology may be used to digitise objects or scenes so that they can be virtually experienced by a customer (e.g. photorealistic online fashion retail).
- Manufacturing: reflectance independent shape acquisition methods will enable contactless measurement on objects with complex reflectance properties (e.g. metallic surfaces) which are common in this industry. This will inspire novel technology for metrology or visual inspection.
- Robotics: the research will lead to advances in sensing technology useful in robot vision to improve the reliability and safety of mobile robot applications by providing richer representations of the environment (including for example information on material properties).
- Medicine: Human tissues have complex reflectance properties which prevent use of traditional image-based modelling techniques. The proposed research has the potential to result in novel non invasive tools for clinical diagnosis or surgical assistance.
Exploitation in these areas will be explored in the context of parallel or subsequent projects using the expertise in CVSSP and the University and links to the industry to engage with the relevant communities.
The creative industry is a rapidly evolving sector with a constant demand for novel technologies to improve the quality of content and reduce production times and costs for competitiveness. Currently the lack of suitable techniques for capture of scenes with complex surface reflectance properties is problematic as it places a tremendous burden on CG artists who need to manually generate photorealistic models. This increases post production times and costs and prevents novel applications due to budget constraints (e.g. limited photorealism in games or online fashion retail). This proposal is focused on achieving the fundamental scientific advances required to enable practical technologies for capture of dynamic shape with arbitrary surface reflectance. This will enable the generation of more sophisticated visual effects by removing current limitations on captured scenes with potential to significantly cut down production times and costs via reduction of the amount of manual interaction needed. For example, using current technology it is not possible to capture the performance of a fashion model or actress wearing a garment with complex reflectance properties such as silk to create a digital double. The project will combine CVSSP's expertise in computer vision and reconstruction with the expertise of world-leading companies in film VFX production (Double Negative) and film post-production tools (The Foundry). This will provide the platform to both deploy the technology for evaluation in production and enable technology transfer within the leading post-production tools (Nuke) for evaluation by production professionals. This will allow early prototyping and trialling of the technology to identify commercialisation opportunities.
The project will have a wide impact beyond the creative industries including the following communities (see further details in Pathways to Impact):
- Cultural heritage: the research will result in photorealistic digitisation techniques allowing preservation of cultural assets via creation of digital copies and supporting cultural enrichment of society by facilitating dissemination and access to cultural assets via the internet. Cultural assets considered will include artefacts held in museums and dynamic content such as traditional dances or performances.
- Online retail: the technology may be used to digitise objects or scenes so that they can be virtually experienced by a customer (e.g. photorealistic online fashion retail).
- Manufacturing: reflectance independent shape acquisition methods will enable contactless measurement on objects with complex reflectance properties (e.g. metallic surfaces) which are common in this industry. This will inspire novel technology for metrology or visual inspection.
- Robotics: the research will lead to advances in sensing technology useful in robot vision to improve the reliability and safety of mobile robot applications by providing richer representations of the environment (including for example information on material properties).
- Medicine: Human tissues have complex reflectance properties which prevent use of traditional image-based modelling techniques. The proposed research has the potential to result in novel non invasive tools for clinical diagnosis or surgical assistance.
Exploitation in these areas will be explored in the context of parallel or subsequent projects using the expertise in CVSSP and the University and links to the industry to engage with the relevant communities.
Publications
Addari G
(2019)
An MRF Optimisation Framework for Full 3D Helmholtz Stereopsis
Addari G
(2023)
A Family of Approaches for Full 3D Reconstruction of Objects with Complex Surface Reflectance
in International Journal of Computer Vision
Bailey M
(2022)
Finite Aperture Stereo
in International Journal of Computer Vision
Roubtsova N
(2017)
Colour Helmholtz Stereopsis for Reconstruction of Dynamic Scenes with Arbitrary Unknown Reflectance.
in International journal of computer vision
Roubtsova N
(2018)
Bayesian Helmholtz Stereopsis with Integrability Prior.
in IEEE transactions on pattern analysis and machine intelligence
Roubtsova N.
(2016)
Decoupled Shape and Appearance Acquisition for Photometrically Complex Scenes
Roubtsova Nadejda S.
(2016)
Accurate 3D reconstruction of dynamic scenes with complex reflectance properties
Description | The project has pioneered techniques for modelling dynamic scenes with an emphasis on scenes with complex material properties (e.g. glossy materials) which are notoriously difficult to model. Key findings include: - A new methodology for modelling dynamic scenes with arbitrary material properties. The approach developed is agnostic to the scene's surface reflectance properties and is therefore applicable to a significantly broader class of scenes than traditional methods. The approach estimates both the shape (depth and surface normals) and the surface material properties (BRDF). - A novel optimisation approach to enforce the coherence of depth and surface normals during reconstruction. The approach was found to significantly improve modelling accuracy and robustness. - A flexible acquisition setup requiring only three cameras and three light sources. The design uniquely combines collocation of cameras and light sources with wavelength multiplexing to enable digitisation of dynamic scenes from a minimal number of cameras and light sources. - An evaluation of the technology on a range of static and dynamic scenes captured during the project. The evaluation considers the modelling accuracy and the application to the production of visual effects such as scene relighting. This resulted in the release of two new datasets to the scientific community. |
Exploitation Route | The research findings open up new capabilities in terms of modelling natural scenes. This is of direct relevance to researchers in computer vision and graphics who are concerned with accurate scene modelling. Additionally, the research has impact in other fields concerned with scene analysis and understanding such as robotics or machine learning. The proposed optimisation approach is of high relevance to researchers working on photometric reconstruction methods (shape from shading, photometric stereo) where it can be used as an alternative to error-prone normal integration. The research findings will also benefit non-academic practitioners with the creative industry being the primary beneficiary. In the creative industry, this technology opens up the possibility to digitise assets for applications such as film post-production, design and gaming. The technology may find applications in other sectors where modelling scenes with complex reflectance properties is required such as in medicine, cultural heritage, retail, manufacturing, etc. |
Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Healthcare Manufacturing including Industrial Biotechology Culture Heritage Museums and Collections |
Description | Research has introduced methods for modelling scenes with complex materials such a glossy surfaces. The technology has been used to digitise a number of complex objects as well as people. Results were shared with industrial partners working in the creative industry to identify possible applications in this industry. The technology extends the class of scenes that can be digitised as most natural and man made scenes have a non-trivial reflectance. This could open up the possibility to create more realistic digital copies of objects and scenes. |
First Year Of Impact | 2016 |
Sector | Creative Economy,Digital/Communication/Information Technologies (including Software) |
Impact Types | Cultural Societal Economic |
Description | Multi-view computational scene modelling |
Amount | £0 (GBP) |
Funding ID | 1815219 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2016 |
End | 09/2020 |
Description | Photorealistic Digitisation and Rendering of Scenes with Complex Materials |
Amount | £0 (GBP) |
Funding ID | 2114902 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2018 |
End | 09/2022 |
Description | Royal Society Research Grant |
Amount | £14,911 (GBP) |
Funding ID | RG150625 |
Organisation | The Royal Society |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 03/2016 |
End | 03/2017 |
Title | Colour Helmholtz Stereopsis Dataset |
Description | The dataset contains a range of static and dynamic scenes, each captured simultaneously by 3 Viper cameras under multi-spectral (RGB) illumination and featuring objects with reflectance properties of varying complexity. The intended use of the data is for geometric 3D reconstruction by Colour Helmholtz Stereopsis. Additional data essential for geometric and photometric calibration procedures as well as the pre-computed calibration files are also included. |
Type Of Material | Database/Collection of data |
Year Produced | 2016 |
Provided To Others? | Yes |
Impact | This is the first dataset dedicated to the reconstruction of scenes using colour Helmholtz stereopsis. The dataset is made publicly available to the research community for non-commercial use. It is anticipated that the public release of this dataset will facilitate the development and evaluation of new algorithms aimed at reconstructing scenes with complex surface reflectance. |
URL | http://www.cvssp.org/data/colourhs/ |
Title | Finite Aperture Stereo Datasets |
Description | This landing page contains the datasets presented in the paper "Finite Aperture Stereo". The datasets are intended for defocus-based 3D reconstruction and analysis. Each download link contains images of a static scene, captured from multiple viewpoints and with different focus settings. The captured objects exhibit a range of reflectance properties and are physically small in scale. Calibration images are also available. |
Type Of Material | Database/Collection of data |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | This dataset enables benchmarking of 3D reconstruction algorithms. |
URL | https://openresearch.surrey.ac.uk/esploro/outputs/dataset/99661466602346 |
Title | Helmholtz Stereopsis Synthetic Dataset |
Description | The dataset consists of synthetic images for three test objects intended to be used as a benchmark for reconstruction via Helmholtz Stereopsis. This also includes ground truth data for quantitative evaluation. |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This dataset provides a benchmark to measure the performance of Helmholtz stereopsis reconstruction algorithms. |
URL | http://www.cvssp.org/data/bayesianhs/ |
Description | Foundry/Double Negative |
Organisation | Double Negative |
Country | United Kingdom |
Sector | Private |
PI Contribution | Developed novel approaches for the digitisation of scenes with complex surface reflectance. |
Collaborator Contribution | In kind contribution. Provided feedback on the research, advised on the suitability for use in creative industries and provided licenses to software. |
Impact | The collaboration has resulted in the development of a prototype acquisition system at the University of Surrey and testing on a range of static and dynamic scenes. |
Start Year | 2015 |
Description | Foundry/Double Negative |
Organisation | The Foundry Visionmongers Ltd |
Country | United Kingdom |
Sector | Private |
PI Contribution | Developed novel approaches for the digitisation of scenes with complex surface reflectance. |
Collaborator Contribution | In kind contribution. Provided feedback on the research, advised on the suitability for use in creative industries and provided licenses to software. |
Impact | The collaboration has resulted in the development of a prototype acquisition system at the University of Surrey and testing on a range of static and dynamic scenes. |
Start Year | 2015 |
Description | BMVA symposium on Dynamic Scene Reconstruction, London |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | This one day symposium co-organised by Dr Marco Volino, Dr Armin Mustafa and Dr Jean-Yves Guillemaut was attended by over 60 people from academia and industry. The event included keynotes, talks, posters and demos from international experts who presented the latests developments on dynamic scene modelling and shared their perspectives during a panel session. As part of the event, Dr Nadejda Roubtsova gave a talk on "Colour Helmholtz Stereopsis: Modelling Dynamic Scenes with Arbitrary Unknown Reflectance Properties" to present the research conducted in the project. |
Year(s) Of Engagement Activity | 2017 |
Description | Presentation at European Conference on Visual Media Production (CVMP) in London |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Presentation of the research and its applications in the form of a poster at European Conference on Visual Media Production which has a large representation from creative industries. |
Year(s) Of Engagement Activity | 2016 |