Shape and Reflectance Acquisition of Complex Dynamic Scenes

Scene modelling is central to many applications in our society including quality control in manufacturing, robotics, medical imaging, visual effects production, cultural heritage and computer games. It requires accurate estimation of the scene's shape (its 3D surface geometry) and reflectance (how its surface reflects light). However, there is currently no method capable of capturing the shape and reflectance of dynamic scenes with complex surface reflectance (e.g. glossy surfaces). This lack of generic methods is problematic as it limits the applicability of existing techniques to scene categories which are not representative of the complexity of natural scenes and materials. This project will introduce a general framework to enable the capture of shape and reflectance of complex dynamic scenes thereby addressing an important gap in the field.

Current image or video-based shape estimation techniques rely on the assumption that the scene's surface reflectance is diffuse (it reflects light uniformly in all directions) or assume it is known a priori thus limiting the applicatibility to simple scenes. Reflectance estimation requires estimation of a 6-dimensional function (the BRDF) which describes how light is reflected at each surface point as a function of incident light direction and viewpoint direction. Due to high dimensionality, reflectance estimation remains limited to static scenes or requires use of expensive specialist equipment. At present, there is no method capable of accurately capturing both shape and reflectance of general dynamic scenes, yet scenes with complex unknown reflectance properties are omnipresent in our daily lives.

The proposed research will address this gap by introducing a novel framework which enables estimation of shape and reflectance for arbitrary dynamic scenes. The approach is based on two key scientific advances which tackle the high dimensionality issue of shape and reflectance estimation. First, a general methodology for decoupling shape estimation from reflectance estimation will be proposed; this will allow decomposition of the original high dimensional problem, which is ill-posed, into smaller sub-problems that are tractable. Second, a space-time formulation of reflectance estimation will be introduced; this will utilise dense surface tracking techniques to extend reflectance estimation to the temporal domain and thereby increase the number of observations available to overcome the inherently low number of observations at a single time instant. This will build on the PI's pioneering research in 3D reconstruction of scenes with arbitrary unknown reflectance properties and his expertise in dynamic scene reconstruction, surface tracking/animation and reflectance estimation.

This research represents a radical shift in scene modelling which will result in several major technical contributions: 1) a reflectance independent shape estimation methodology for dynamic scenes, 2) a non-rigid surface tracking method suitable for general scenes with complex and unknown reflectance and 3) a general and scalable reflectance estimation method for dynamic scenes. This will benefit all areas requiring accurate acquisition of shape and reflectance for real-world scenes with complex dynamic shape and reflectance without the requirement for complex and restrictive hardware setups; such scenes are a common occurrence in natural environments, manufacturing (metallic surfaces) and medical imaging (human tissue) but accurate capture of shape is not possible with existing approaches which assume diffuse reflectance and fail dramatically for such cases. This will achieve for the first time accurate modelling of dynamic scenes with arbitrary surface reflectance properties thus opening up novel avenues in scene modelling. The application of this technology will be demonstrated in digital cinema in collaboration with industrial partners to support the development of the next generation of visual effects.

The proposed research will have significant impact in the creative industry, particularly the visual media production community (film, television, game) where accurate shape and reflectance estimation are critical to produce photorealistic visual effects or digital characters as well as other communities where shape and reflectance estimation of scenes with complex surface properties are beneficial (cultural heritage, retail, manufacturing, robotics and medicine).

The creative industry is a rapidly evolving sector with a constant demand for novel technologies to improve the quality of content and reduce production times and costs for competitiveness. Currently the lack of suitable techniques for capture of scenes with complex surface reflectance properties is problematic as it places a tremendous burden on CG artists who need to manually generate photorealistic models. This increases post production times and costs and prevents novel applications due to budget constraints (e.g. limited photorealism in games or online fashion retail). This proposal is focused on achieving the fundamental scientific advances required to enable practical technologies for capture of dynamic shape with arbitrary surface reflectance. This will enable the generation of more sophisticated visual effects by removing current limitations on captured scenes with potential to significantly cut down production times and costs via reduction of the amount of manual interaction needed. For example, using current technology it is not possible to capture the performance of a fashion model or actress wearing a garment with complex reflectance properties such as silk to create a digital double. The project will combine CVSSP's expertise in computer vision and reconstruction with the expertise of world-leading companies in film VFX production (Double Negative) and film post-production tools (The Foundry). This will provide the platform to both deploy the technology for evaluation in production and enable technology transfer within the leading post-production tools (Nuke) for evaluation by production professionals. This will allow early prototyping and trialling of the technology to identify commercialisation opportunities.

The project will have a wide impact beyond the creative industries including the following communities (see further details in Pathways to Impact):
- Cultural heritage: the research will result in photorealistic digitisation techniques allowing preservation of cultural assets via creation of digital copies and supporting cultural enrichment of society by facilitating dissemination and access to cultural assets via the internet. Cultural assets considered will include artefacts held in museums and dynamic content such as traditional dances or performances.
- Online retail: the technology may be used to digitise objects or scenes so that they can be virtually experienced by a customer (e.g. photorealistic online fashion retail).
- Manufacturing: reflectance independent shape acquisition methods will enable contactless measurement on objects with complex reflectance properties (e.g. metallic surfaces) which are common in this industry. This will inspire novel technology for metrology or visual inspection.
- Robotics: the research will lead to advances in sensing technology useful in robot vision to improve the reliability and safety of mobile robot applications by providing richer representations of the environment (including for example information on material properties).
- Medicine: Human tissues have complex reflectance properties which prevent use of traditional image-based modelling techniques. The proposed research has the potential to result in novel non invasive tools for clinical diagnosis or surgical assistance.
Exploitation in these areas will be explored in the context of parallel or subsequent projects using the expertise in CVSSP and the University and links to the industry to engage with the relevant communities.