Photorealistic Digitisation and Rendering of Scenes with Complex Materials
Lead Research Organisation:
University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP
Abstract
Traditional scene modelling methods assume a simple or a known reflectance model and are usually unable to deal with sub-surface scattering, a phenomenon commonly encountered in natural and man-made materials. Besides, they often rely on controlled capture conditions to achieve high accuracy. This research will address these limitations by generalising scene modelling to scenes exhibiting complex reflectance including sub-surface scattering. This will be achieved by leveraging recent advances in computational photography and machine learning to capture a richer scene representation of both shape and appearance. Integration of different cues such as stereo and focus information with priors learnt using deep neural networks will provide a framework to resolve traditional ambiguities and improve modelling accuracy. In turn, the research will enable the creation of more photorealistic digital copies of people and objects and will remove traditional limitations on acquisition to enable scene modelling in uncontrolled capture conditions.
Organisations
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509772/1 | 01/10/2016 | 30/09/2021 | |||
| 2114902 | Studentship | EP/N509772/1 | 01/10/2018 | 31/03/2022 | MATTHEW BAILEY |
| Description | Defocus formation from a finite aperture is a well-known phenomena, occurring in many forms of photographic media. Although often exploited for artistic reasons, a surprising amount of information about the scene structure is encoded in the camera's blurring function. The aim of this work is to explore how this can be leveraged to improve the recovery of 3D geometry from scenes with complex reflectance. The first contribution of this work aims to characterise the camera response to out-of-focus regions of the image. 'Depth from Defocus' (DFD) is a well-established field that aims to utilise this information to reconstruct scene geometry from analysis of out-of-focus regions, usually modelling the camera as a thin lens for simplicity. While many existing methods achieve approximate depth maps suitable for some applications, the majority are limited to geometrically inconsistent single-view reconstructions. Those which attempt the integration of additional viewpoints typically rely on iterative parameter refinement, and generally fail to significantly improve surface resolution. A novel contribution of our work is the recovery of complete 3D reconstructions using only defocus information. We achieve this by adopting a thick lens camera model, which better describes the camera's defocus response. To do so required the development of a practical thick lens camera calibration, which was used to capture several datasets. We found our approach significantly outperforms the traditional thin lens model, both in a single-view and multi-view context. The second contribution looks at the complementary properties of defocus and stereo reconstruction cues. Unlike conventional multi-view stereo (MVS), which depends on photo-metric consistency between views, DFD requires only a single viewpoint for reconstruction. This makes defocus-based approaches naturally robust to view-dependent materials considered challenging for traditional MVS. Conversely, textures which are invariant to defocus can be suitable for correspondence. We investigated this complementary relationship to determine the benefits of combining defocus information with stereo cues. Our results demonstrated an improvement over DFD alone despite the specular and reflective nature of the datasets, and typically outperformed modern MVS. The third on-going contribution explores the novel application of neural rendering to defocus modelling. The key aspects of the defocus image formation model described by a focal stack (depth, radiance and point spread function) are disentangled by modelling each component as a multi-layer perception network. Novel refocused images can be rendered that accurately capture the bokeh produced by specular highlights with arbitrary aperture shapes and diameters. |
| Exploitation Route | The thick lens calibration we developed makes capturing calibrated focal stacks practical for other researchers without requiring specialist or custom equipment. Our reconstruction approach is capable of recovering the geometry of macro-scale scenes that was previously not possible with conventional methods. We intend to make our datasets publicly available to help advance research in this area further. |
| Sectors | Digital/Communication/Information Technologies (including Software),Other |