Human Vision: Relationship to Three-Dimensional Surface Statistics of Natural Scenes

The human visual system has been fine-tuned over generations of evolution to operate effectively in our particular environment, allowing us to form rich 3D representations of the objects around us. The scenes that we encounter on a daily basis produce 2D retinal images that are complex and ambiguous. From this input, how does the visual system achieve the immensely difficult goal of recovering our surroundings, in such an impressively fast and robust way?
To achieve this feat, humans must use two types of information about their environment. First, we must learn the probabilistic relationships between 3D natural scene properties and the 2D image cues these produce. Second, we must learn which scene structures (shapes, distances, orientations) are most common, or probable in our 3D environment. This statistical knowledge about natural 3D scenes and their projected images allows us to maximize our perceptual performance. To better understand 3D perception, therefore, we must study the environment that we have evolved to process. A key goal of our research is to catalogue and evaluate the statistical structure of the environment that guides human depth perception. We will sample the range of scenes that humans frequently encounter (indoor and outdoor environments over different seasons and lighting conditions). For each scene, state-of-the-art ground based Light Detection and Ranging (LiDAR) technology will be used to measure the physical distance to all objects (trees, ground, etc.) from a single location - a 3D map of the scene. We will also take High Dynamic Range (HDR) photographs of the same scene, from the same vantage point. By collating this paired 3D and 2D data across numerous scenes we will create a comprehensive database of our environment, and the 2D images that it produces. By making the database publicly available it will facilitate not just our own work, but research by human and computer vision scientists around the world who are interested in a range of pure and applied visual processes.
There is great potential for computer vision to learn from the expert processor that is the human visual system: computer vision algorithms are easily out-performed by humans for a range of tasks, particularly when images correspond to more complex, realistic scenes. We are still far from understanding how the human visual system handles the kind of complex natural imagery that defeats computer vision algorithms. However, the robustness of the human visual system appears to hinge on: 1) exploiting the full range of available depth cues and 2) incorporating statistical 'priors': information about typical scene configurations. We will employ psychophysical experiments, guided by our analyses of natural scenes and their images, to develop valid and comprehensive computational models of human depth perception. We will concentrate our analysis and experimentation on key tasks in the process of recovering scene structure - estimating the location, orientation and curvature of surface segments across the environment. Our project addresses the need for more complex and ecologically valid models of human perception by studying how the brain implicitly encodes and interprets depth information to guide 3D perception.
Virtual 3D environments are now used in a range of settings, such as flight simulation and training systems, rehabilitation technologies, gaming, 3D movies and special effects. Perceptual biases are particularly influential when visual input is degraded, as they are in some of these simulated environments. To evaluate and improve these technologies we require a better understanding of 3D perception. In addition, the statistical models and inferential algorithms developed in the project will facilitate the development of computer vision algorithms for automatic estimation of depth structure in natural scenes. These algorithms have many applications, such as 2D to 3D film conversion, visual surveillance and biometrics.

Our proposed work sits at the interface of human and computer vision. In essence, it asks how humans and computers infer 3D structure from 2D images in realistic, complex environments. Beyond these academic arenas, our work has clear implications for those working in applied computer vision and in the visual media industry. The latter two groups will exploit our work in technologies such as 2D to 3D conversion, special effects generation and creating virtual reality environments for applications such as gaming and training. The recent NextGen Review (2011) of the skill requirements (and current shortfall) for the UK's video games and visual effects industries highlighted the importance of those industries to the UK economy. In 2008, the global sales of video games created by UK companies reached £2 billion, contributing £1 billion in GDP, making the UK the third largest games developer in the world. The UK visual effects industry is also on the rise, contributing to blockbuster movies like Harry Potter, Inception and Batman. This sector grew by 17% between 2006 and 2008, with four of the worlds largest visual effects companies based in London.
Our work has a clear role in maintaining the lead role that the UK currently holds in these growing industries. These world-leading industries could benefit substantially from the input of experts in vision science. For example, many of the industry applications described above involve the inference of 3D scene structure from 2D images, but often have to rely on human hand segmentation and depth labeling of images to complement current computational algorithms. In contrast, humans are remarkably adept and robust in reconstructing their 3D world. Our work will expand current understanding of the structure of natural scenes, and how this statistical structure is exploited by the human visual system to efficiently recover depth. This ecological, natural scenes approach is critical to bridging the gap between human performance and current efforts to replicate it in computer vision applications. To ensure that this impact is realized, vision scientists must engage with those involved in gaming and visual effects. Currently, there is a lack of communication between vision researchers, and these industrial groups. Dr. Adams' discussions with attendees at the recent Conference on Visual Media Production (CVMP) in London made clear the potential for human vision to inform algorithms for visual media production that are efficient, and produce content that is realistic and enjoyable for the end user. For example, certain well-known strategies within human vision for recovering shape from shading are not exploited within technologies that capture and create 3D content. Discussions with our project partners (Hilton: applied computer vision and Grau: 3D media production, BBC) have identified particular areas where our work will inform current problems within applied computer vision and visual media production; the potential impact of our work is reflected in their Letters of Support.
The NextGen report identified an immediate need to change current practice in ICT training in schools and Universities to better reflect the skill needs of the gaming and visual media industries. This move is critical to ensure that these industries continue to lead the world market. Vision science relies heavily on the key skills highlighted in the report, including mathematics, physics, computer programming and design. By using visual illusions to explain key concepts in human vision, and demonstrating the mathematical and computational challenges in developing visual stimuli, we will design activities (for our website and for the science roadshow) that will engage young people and foster an interest in mathematics, perceptual psychology and its applications. By making use of the engaging aspects of visual perception we hope to inspire future generations of scientists and industry professionals.