Realistic Shape from Shading

Shape-from-shading (SFS) is a classic problem in computer vision. It aims to estimate 3D surface shape from the variations in shading in a single photographic image. The fact that it recovers shape using only a single image makes SFS attractive to a wide range of applications, especially when other 3D imaging techniques such as stereo or depth scanners are difficult to apply. Example applications can be found in topography analysis of SAR (synthetic aperture radar) images, reconstruction of medical images, inspection of microelectronics, CAD systems, and the entertainment industry. However, despite over four decades research, SFS still remains a challenging problem which is underused in real world problems, due to a lack of robustness, and sometimes implausible results. A good solution is pressing and challenging. This project intends to develop a robust and practical SFS algorithm for accurate shape recovery from real-world images.

The reasons for SFS's current poor performance on real-world images have several underlying causes. The first is that the classic assumptions of orthographic projection, Lambertian reflection, and simple lighting models are inaccurate for real-world surfaces. The second reason is that SFS is an underconstrained problem: the human visual system recovers shape not only from shading, but also from outlines, shadows, and prior experience. In computer vision, little work has considered the combination of shape from shading with other visual cues and human interactions. The third and largely overlooked reason is that many real surfaces are not smooth, and have detailed features. Most existing SFS algorithms only apply to images of smooth surfaces, and tend to over-smooth any features.

Based on these observations, this project will integrate techniques from such areas as feature-aware image filtering, shape from line drawing, and user interaction, to achieve more accurate shape recovery from sophisticated real-world images. An interactive platform for SFS will be developed for realistic applications. The outcome of the research will be tested on various applications in CAD and computer vision: specifically, as part of the project, we will explore the applications to bas-relief generation, and face recognition.

SFS is a classic problem in computer vision. A reliable SFS algorithm will have applications in many areas, such as manufacturing, security, medicine, entertainment, and advertising. This project aims to develop robust and practical SFS algorithms which can achieve accurate shape recovery from real-world images. A platform for interactive SFS will also be developed. The outcome of the research will be of benefit to both academic and real-world users.

The UK is a world leader in computer vision, in which SFS has remained a classic but challenging topic. This project will extend existing SFS techniques, helping the UK retain its leadership in this area. This will contribute to the world class reputation of UK universities, which are a significant export earner. This project will also gain the research assistant broad knowledge and skills in 3D reconstruction and its application, which will potentially train a highly competent researcher in the future.

This project is aligned with the EPSRC theme of 'Manufacturing the Future'. In manufacturing industry, SFS has potential applications in e.g. aerospace, electronics, and coinage manufacture, which are all important to the UK economy.

Shape-from-shading is an important tool for recovering terrain topography from synthetic aperture radar (SAR) images. However, radar data is notoriously noisy. We will develop SFS algorithms which preserve salient features while smoothing the surface, and have the ability to manipulate the detail at different scales in recovered shapes. The new algorithms could help to separate noise from actual surface structures in SAR images.

In microelectronics inspection, SFS is used to recover wafer profiles from images taken using a scanning electron microscope. It enables 3D surface measurements from 2D SEM images, which is important in the quality control of integrated circuits. The interactive platform developed in this project could provide a way to obtain priori knowledge about the structures of the circuits, and hence improve the accuracy of shape recovery and inspection.

Bas-relief creation is essential but time-consuming in coinage manufacture. The feature preserving ability of the SFS algorithms developed in this project is consistent with the needs of bas-relief creation. Thus, we will explore the application of our work to automatic bas-relief generation from photographs in collaboration with the Royal Mint.

Besides manufacturing, our work will also have benefits to security systems, medical imaging, and daily life.

Face recognition has been utilized in security systems. Current systems use 2D appearance, but in comparison, 3D facial shape cannot be easily modified, and provides more reliable information for surveillance purposes. This project will explore the application of the new algorithms to deriving 3D facial shape from 2D photographs.

In medical imaging, SFS is useful to correct area measurement errors, caused by 2D projection of 3D organs, in endoscopy. The interactive platform provided by this project will potentially enhance use of knowledge of medical practitioners, and further improve the measurement accuracy.

The entertainment industry, including movies and video games, contributes greatly to UK revenues. 3D models are widely used for animation, special effects, 3D movies, and games. Reliable SFS algorithms could permit the construction of a wider range of 3D models by shape recovery from photographs, and improve the efficiency of model creation. Moreover, SFS may provide an alternative way to create 3D films, perhaps from classic 2D movies.

SFS also has potential to impact our daily life, via more sophisticated photo processing software, interior design software, and 3D effects in advertising. Devising realistic SFS algorithms is consistent with the EPSRC theme aiming to 'Rapidly realise the transformational impact of digital technologies on: life, cultural experiences, future society, and the economy'.