Scene Processing with Machine Learnable and Semantically Parametrized Scene Representations

Digital representations of visual reality and imagination is an integral part of almost all scientific disciplines, industry, and society. Imaging techniques and computer graphics have successfully solved the problems of creating accurate projections, e.g. image and video, of the real world over the last decades. However, such projections are fundamentally limited representations of the underlying scenes and only allow for passive consumption such as viewing on a screen. Many applications from image editing to augmented/virtual reality instead require active 3D exploration, creation, and editing of scenes, for which we need full virtual scene models.

Creating virtual scenes that are high fidelity models of the real world and beyond via digital modelling or capture has traditionally been a privilege only available to corporations with educated artists and engineers, working with complex software and hardware tools for countless hours. These professionals produce and work with carefully designed parametrizations of geometry, appearance, and motion, which allow them to author and edit virtual scenes with all the intricate details of reality and their imagination. On the other end, many computer vision techniques try to have a shortcut by capturing scenes from the real world via simpler sensors in uncontrolled environments, or altering images of scenes with algorithms that only implicitly encode scene semantics, e.g. in latent spaces of artificial neural networks. These lead to scene representations of lower quality and in a form that is not easily editable for semantically meaningful modelling.

The objective of this research project is to tackle these shortcomings and develop a scene representation that is 1) a high fidelity detailed model of visual reality in terms of geometry, appearance, and motion, 2) machine learnable via capture in partially controlled practical environments, 3) semantically parametrized to allow for easy and intuitive edits, 4) fast to visualize for real-time exploration. Based on this representation, we will develop scene processing techniques that will allow individuals to create, alter, explore, and share high fidelity virtual objects and scenes, unlocking a completely new set of applications in augmented/virtual reality, gaming, product design, manufacturing, education, robotics, and medical domains.

This project is on digital modelling, capture, and visualization of 3D reality, which has become one of the main driving forces behind the progress in science, industry, and society. Its impact thus extends far beyond the primary scientific disciplines it rests on, computer graphics, computer vision, and machine learning.

The application areas include augmented/ virtual reality (AR/VR), video games, filmmaking, telecommunication, product design, manufacturing, health, robotics, education, and smart cities. In many of these areas, academia and industry are working closely together. Apart from scientific impact, the project thus has the potential to have a significant short and long-term impact on industry, which also extends to the public sector e.g. digital exhibitions and museums, improved healthcare services, or urban planning.

Digital capture and interactive visualizations of scenes have the potential to transform the way many industries are currently run. I have already led several research projects in the creative industries with the long-term goal of transforming the way video games and films are created via novel 3D capture tools, AR/VR displays, and human computer interfaces; and providing immersive 3D experiences for films, games, and telecommunications. Microsoft, Facebook, Apple, and Google are heavily investing in AR/VR devices for similar applications where virtual scenes are created, edited, and explored.

3D digital reality is also transforming product design, where designers can digitally create and optimize products without having to build a physical prototype, and do fast prototyping without going through expensive manufacturing cycles. Accurate digital models with precise overlays on reality via AR devices can then be used to guide manufacturing processes for reduced errors and costs. Such technologies are already attracting massive attention from many industries such as automotive and aviation. The outputs of this project will help to capture, track, and edit 3D objects for these applications.

Healthcare is another major area where 3D scene processing has numerous important applications. AR/VR based systems are being explored for therapy e.g. for mental problems, addictions, phobias, or problems associated with missing limbs. Further use cases are diagnosis by overlaying virtual layers of tissues or organs and showing relevant information, surgery planning, assistance, and remote operations with virtual models of human bodies.

Robotic systems fundamentally rely on building digital models of environments to navigate and perform tasks. All robotics applications will thus fundamentally benefit from the developed scene representations and techniques. High fidelity scene representations can transform human robot interactions by accurate perception of humans and robot-human coordination. Such systems have a diversity of applications such as rescue operations, law enforcement, assisting disabled individuals, or creating artworks.

Editable and high fidelity digital models of reality capturing objects from furniture or buildings to entire cities or nature can also be instrumental for education and smart cities with immersive 3D visualizations for training, AR/VR systems for individuals with disabilities, preserving and visualizing cultural heritage, urban planning, or navigation.