Realistic Shape from Shading
Lead Research Organisation:
Cardiff University
Department Name: Computer Science
Abstract
Shape-from-shading (SFS) is a classic problem in computer vision. It aims to estimate 3D surface shape from the variations in shading in a single photographic image. The fact that it recovers shape using only a single image makes SFS attractive to a wide range of applications, especially when other 3D imaging techniques such as stereo or depth scanners are difficult to apply. Example applications can be found in topography analysis of SAR (synthetic aperture radar) images, reconstruction of medical images, inspection of microelectronics, CAD systems, and the entertainment industry. However, despite over four decades research, SFS still remains a challenging problem which is underused in real world problems, due to a lack of robustness, and sometimes implausible results. A good solution is pressing and challenging. This project intends to develop a robust and practical SFS algorithm for accurate shape recovery from real-world images.
The reasons for SFS's current poor performance on real-world images have several underlying causes. The first is that the classic assumptions of orthographic projection, Lambertian reflection, and simple lighting models are inaccurate for real-world surfaces. The second reason is that SFS is an underconstrained problem: the human visual system recovers shape not only from shading, but also from outlines, shadows, and prior experience. In computer vision, little work has considered the combination of shape from shading with other visual cues and human interactions. The third and largely overlooked reason is that many real surfaces are not smooth, and have detailed features. Most existing SFS algorithms only apply to images of smooth surfaces, and tend to over-smooth any features.
Based on these observations, this project will integrate techniques from such areas as feature-aware image filtering, shape from line drawing, and user interaction, to achieve more accurate shape recovery from sophisticated real-world images. An interactive platform for SFS will be developed for realistic applications. The outcome of the research will be tested on various applications in CAD and computer vision: specifically, as part of the project, we will explore the applications to bas-relief generation, and face recognition.
The reasons for SFS's current poor performance on real-world images have several underlying causes. The first is that the classic assumptions of orthographic projection, Lambertian reflection, and simple lighting models are inaccurate for real-world surfaces. The second reason is that SFS is an underconstrained problem: the human visual system recovers shape not only from shading, but also from outlines, shadows, and prior experience. In computer vision, little work has considered the combination of shape from shading with other visual cues and human interactions. The third and largely overlooked reason is that many real surfaces are not smooth, and have detailed features. Most existing SFS algorithms only apply to images of smooth surfaces, and tend to over-smooth any features.
Based on these observations, this project will integrate techniques from such areas as feature-aware image filtering, shape from line drawing, and user interaction, to achieve more accurate shape recovery from sophisticated real-world images. An interactive platform for SFS will be developed for realistic applications. The outcome of the research will be tested on various applications in CAD and computer vision: specifically, as part of the project, we will explore the applications to bas-relief generation, and face recognition.
Planned Impact
SFS is a classic problem in computer vision. A reliable SFS algorithm will have applications in many areas, such as manufacturing, security, medicine, entertainment, and advertising. This project aims to develop robust and practical SFS algorithms which can achieve accurate shape recovery from real-world images. A platform for interactive SFS will also be developed. The outcome of the research will be of benefit to both academic and real-world users.
The UK is a world leader in computer vision, in which SFS has remained a classic but challenging topic. This project will extend existing SFS techniques, helping the UK retain its leadership in this area. This will contribute to the world class reputation of UK universities, which are a significant export earner. This project will also gain the research assistant broad knowledge and skills in 3D reconstruction and its application, which will potentially train a highly competent researcher in the future.
This project is aligned with the EPSRC theme of 'Manufacturing the Future'. In manufacturing industry, SFS has potential applications in e.g. aerospace, electronics, and coinage manufacture, which are all important to the UK economy.
Shape-from-shading is an important tool for recovering terrain topography from synthetic aperture radar (SAR) images. However, radar data is notoriously noisy. We will develop SFS algorithms which preserve salient features while smoothing the surface, and have the ability to manipulate the detail at different scales in recovered shapes. The new algorithms could help to separate noise from actual surface structures in SAR images.
In microelectronics inspection, SFS is used to recover wafer profiles from images taken using a scanning electron microscope. It enables 3D surface measurements from 2D SEM images, which is important in the quality control of integrated circuits. The interactive platform developed in this project could provide a way to obtain priori knowledge about the structures of the circuits, and hence improve the accuracy of shape recovery and inspection.
Bas-relief creation is essential but time-consuming in coinage manufacture. The feature preserving ability of the SFS algorithms developed in this project is consistent with the needs of bas-relief creation. Thus, we will explore the application of our work to automatic bas-relief generation from photographs in collaboration with the Royal Mint.
Besides manufacturing, our work will also have benefits to security systems, medical imaging, and daily life.
Face recognition has been utilized in security systems. Current systems use 2D appearance, but in comparison, 3D facial shape cannot be easily modified, and provides more reliable information for surveillance purposes. This project will explore the application of the new algorithms to deriving 3D facial shape from 2D photographs.
In medical imaging, SFS is useful to correct area measurement errors, caused by 2D projection of 3D organs, in endoscopy. The interactive platform provided by this project will potentially enhance use of knowledge of medical practitioners, and further improve the measurement accuracy.
The entertainment industry, including movies and video games, contributes greatly to UK revenues. 3D models are widely used for animation, special effects, 3D movies, and games. Reliable SFS algorithms could permit the construction of a wider range of 3D models by shape recovery from photographs, and improve the efficiency of model creation. Moreover, SFS may provide an alternative way to create 3D films, perhaps from classic 2D movies.
SFS also has potential to impact our daily life, via more sophisticated photo processing software, interior design software, and 3D effects in advertising. Devising realistic SFS algorithms is consistent with the EPSRC theme aiming to 'Rapidly realise the transformational impact of digital technologies on: life, cultural experiences, future society, and the economy'.
The UK is a world leader in computer vision, in which SFS has remained a classic but challenging topic. This project will extend existing SFS techniques, helping the UK retain its leadership in this area. This will contribute to the world class reputation of UK universities, which are a significant export earner. This project will also gain the research assistant broad knowledge and skills in 3D reconstruction and its application, which will potentially train a highly competent researcher in the future.
This project is aligned with the EPSRC theme of 'Manufacturing the Future'. In manufacturing industry, SFS has potential applications in e.g. aerospace, electronics, and coinage manufacture, which are all important to the UK economy.
Shape-from-shading is an important tool for recovering terrain topography from synthetic aperture radar (SAR) images. However, radar data is notoriously noisy. We will develop SFS algorithms which preserve salient features while smoothing the surface, and have the ability to manipulate the detail at different scales in recovered shapes. The new algorithms could help to separate noise from actual surface structures in SAR images.
In microelectronics inspection, SFS is used to recover wafer profiles from images taken using a scanning electron microscope. It enables 3D surface measurements from 2D SEM images, which is important in the quality control of integrated circuits. The interactive platform developed in this project could provide a way to obtain priori knowledge about the structures of the circuits, and hence improve the accuracy of shape recovery and inspection.
Bas-relief creation is essential but time-consuming in coinage manufacture. The feature preserving ability of the SFS algorithms developed in this project is consistent with the needs of bas-relief creation. Thus, we will explore the application of our work to automatic bas-relief generation from photographs in collaboration with the Royal Mint.
Besides manufacturing, our work will also have benefits to security systems, medical imaging, and daily life.
Face recognition has been utilized in security systems. Current systems use 2D appearance, but in comparison, 3D facial shape cannot be easily modified, and provides more reliable information for surveillance purposes. This project will explore the application of the new algorithms to deriving 3D facial shape from 2D photographs.
In medical imaging, SFS is useful to correct area measurement errors, caused by 2D projection of 3D organs, in endoscopy. The interactive platform provided by this project will potentially enhance use of knowledge of medical practitioners, and further improve the measurement accuracy.
The entertainment industry, including movies and video games, contributes greatly to UK revenues. 3D models are widely used for animation, special effects, 3D movies, and games. Reliable SFS algorithms could permit the construction of a wider range of 3D models by shape recovery from photographs, and improve the efficiency of model creation. Moreover, SFS may provide an alternative way to create 3D films, perhaps from classic 2D movies.
SFS also has potential to impact our daily life, via more sophisticated photo processing software, interior design software, and 3D effects in advertising. Devising realistic SFS algorithms is consistent with the EPSRC theme aiming to 'Rapidly realise the transformational impact of digital technologies on: life, cultural experiences, future society, and the economy'.
Publications
![publication icon](/resources/img/placeholder-60x60.png)
Wu J
(2016)
Improving Shape from Shading with Interactive Tabu Search
in Journal of Computer Science and Technology
![publication icon](/resources/img/placeholder-60x60.png)
Wu J
(2014)
Use of non-photorealistic rendering and photometric stereo in making bas-reliefs from photographs
in Graphical Models
Description | We first explored new regularizations informed by various edge-aware image filtering techniques in optimisation based shape-from-shading (SFS). We found that new regularizations with edge-aware filtering have better performance than the traditional regularization in handling added noise and blurriness, but do not improve the overall shape recovered. The inherent ambiguities in SFS are the main causes of dramatic distortions in the overall shape reconstruction. Based on this finding, we shifted our attention to focus on solving ambiguities in SFS. We examined various SFS approaches and found that the well-known concave-convex ambiguity is introduced from the initialization step. Based on this finding, we proposed an interactive approach based on heuristic search to allow the user to explore the space of candidate initializations, and demonstrated the intuitiveness of the proposed method compared to existing interactive SFS methods. We then further the research into improving the efficiency of user interaction by making use of belief-propagation to recommend most likely solutions. We developed an interactive platform to support 1) both interactive approaches to solve the concave-convex ambiguity from initialization, and 2) the interactive correction of bas-relief ambiguity as a post-processing step. |
Exploitation Route | We will continue the uncompleted objective to build applications based on the developed algorithms and framework to demonstrate their use in real-world applications. We will continue disseminating our findings and demonstrating the applications in conferences, seminars, and some academic-industrial events to increase the visibility of our work. Demos and source code will be made available online. |
Sectors | Creative Economy,Leisure Activities, including Sports, Recreation and Tourism,Manufacturing, including Industrial Biotechology,Culture, Heritage, Museums and Collections |
URL | http://users.cs.cf.ac.uk/J.Wu/ |
Description | A 7-year-old boy with no knowledge and training in 3D modelling and surface normals, which are otherwise required by other interactive methods to reconstruct 3D shapes from images, successfully used the developed interactive SFS platform to get the 3D shapes of his toys from photos. This demonstrates the usability of our work to a wider user population, and potential impact to the education for the younger population. |
First Year Of Impact | 2016 |
Sector | Creative Economy,Digital/Communication/Information Technologies (including Software) |
Impact Types | Economic |