Acquiring Complete and Editable Outdoor Models from Video and Images

Lead Research Organisation: University of Bath
Department Name: Computer Science

Abstract

Imagine being able to take a camera out of doors and use it to capture 3D models of the world around you. The landscape at large, including valleys and hills replete with trees, rivers, waterfalls, fields of grass, clouds; seasides with waves rolling onto shore here and crashing onto rocks over there; urban environments complete with incidentals such as lamposts, balconies, and the detritus of modern life. Imagine models that look and move like the real thing. Models that you can use with to make up new scenes of your own, which you can control as you please, and render in how you like. You can zoom into to see details, and out to get a wide impression.

This is an impressive vision, and one that is well beyond current know-how. Our plan is to take a major step towards meeting this vision. We will enable users to use video and images to capture large scale scenes of selected types and populate them with models trees, fountains, street furniture and such like, again carefully selecting the types of objects. We will provide software that recognises the sort of environment the camera is in, and objects in that environment, so that 3D moving models can be automatically created.

This will prove very useful to our intended user group, which is the creative industries in the UK: films, games, broadcast. Modelling outdoor scenes is expensive and time consuming, and the industry recognises that video and images are excellent sources for making models they can use. To help them further we will develop software that makes use of their current practice of acquiring survey shots of scenes, so that all data is used at many levels of detail. Finally we will wrap all of our developments into a single system that shows the acquisition, editing and control of complete outdoor environments is one step closer.

Planned Impact

The impact of our work is likely to be significant both academically and industrially. Indeed, we hope to place UK PLC in a world
leading position with respect to model acquisition in outdoor environments.

TO make academic impact we will publish in high quality forums (eg. ICCV, SIGGRAPH) we will organise workshops, two domestically and one internationally. One domestic workshop will be run with the help of the British Machine Vision Association as part of their one-day seminar series, the other via the London Graphics Seminar series which is well attended by both academia and the creative industries. Internationally, we will organise a workshop at a major conference such as Eurographics or ICCV.

Our industrial and communication impact plan has three main parts:
(1) Engage in research of the highest academic quality, yet ensure output are useful to industry; we have convened an
Industrial Advisory Panel (IAP) for this.
(2) Transfer this research into the industry, via the Engineering Doctoral (Eng.D.) centres.
(3) Disseminate our work to the general public as widely as possible.

The IAP keeps our research industrially relevant, without compromising academic integrity. One role of the IAP is to suggest classes of landscape, phenomena of interest and such like, which helps us in selecting which scenes and scene elements to target for model acquisition. A second role is to advise us on industry practice regarding current acquisition via cameras, and modelling needs in terms of editing requirements etc. A third role is to keep us abreast of innovations in industry. The IAP comprises BBC, Disney, Crytec, Gobo Games, EA, Double Negative, and The Foundry, which represents a cross section of the UK's world-class creative sector.

Each academic partner has direct access to an Eng.D. centre: the Centre for Digital Entertainment and Bath, and the Virtual Environments, Imaging and Visualization centre at UCL. Both Eng.D. centres are focussed upon placing doctoral level students into the creative industries, jointly supervised by an academic and an industrial researcher. This makes them the ideal vehicle by which to transfer our IP. We already have working relationships will all IAP members through these Eng.D. centres.

Dissemination to the wider public is important. It helps science in general, and the EPSRC in particular, maintain a relationship with the tax payer. We are luck in that our work is visual in nature and therefore easily accessible at a consumer level - indeed our previous work has enjoyed media reportage on a world wide basis on websites, newpapers, radio and TV. This has resulted in many enquiries from industry and from fellow academics. We will continue to produce show-reels, maintain web-sites and make press releases. In addition we will speak about our work at public meeting, such as Bath Camp.

Publications

10 25 50
publication icon
Beale D (2016) Fitting quadrics with a Bayesian prior in Computational Visual Media

publication icon
Haines T (2016) My Text in Your Handwriting in ACM Transactions on Graphics

publication icon
Hedman P (2016) Scalable inside-out image-based rendering in ACM Transactions on Graphics

publication icon
Hedman P (2018) Deep blending for free-viewpoint image-based rendering in ACM Transactions on Graphics

publication icon
Intharah T (2018) RecurBot

publication icon
Kuo M (2016) Feature-Aware Pixel Art Animation in Computer Graphics Forum

 
Description We are actively engaged with companies to transfer our IP to them. This includes pathways via the Center for Digital Entertainment DTC at Bath / Bournemouth. We have engaged with the public, in particular A-level students as part of a wider dissemination brief. -- update -- We are now working with commerical partners to develop an AR system that works in real-time on mobile devices. The system will be deployed inside commerical stores, that are dynamic (people ,move etc) and which will include difficult elements to deal with. The experience gained in this project will be invaluable in making progress with this commercial project (which is under and NDA). The economici impact is expected to be significant.
First Year Of Impact 2019
Sector Creative Economy,Education,Retail
Impact Types Cultural,Economic

 
Description EPSRC Impact Acceleration Award
Amount £30,000 (GBP)
Funding ID M.2.35 
Organisation University College London 
Department Innovation and Enterprise
Sector Academic/University
Country United Kingdom
Start 11/2016 
End 03/2017
 
Description Impact Studentship, Change of Paradigm
Amount £42,000 (GBP)
Organisation Changing Paradigms, LLC 
Sector Private
Country United States
Start 09/2015 
End 09/2018
 
Description Appearance Capture with Aalto and NVIDIA 
Organisation Aalto University
Department Department of Computer Science
Country Finland 
Sector Academic/University 
PI Contribution In joint research on practical appearance capture, I contributed with my general background in the field: within our team, I was the senior domain expert on appearance capture. Through weekly project meetings with Prof. Lehtinen (Aalto University, NVIDIA Research) and his graduate student Miika Aittala, we jointly pushed forward Mr. Aittala's research agenda, leading to publications at the top venue in our field. Later in the collaboration, there were additional synergies with my EPSRC-funded research on landscape appearance acquisition.
Collaborator Contribution Prof. Lehtinen, who has ample experience in industrial computer graphics research, specialising in rendering and natural light and reflectance phenomena, contributed both through his significant technical and scientific background, but also through the direct supervision of his graduate student Miika Aittala, who himself worked on material appearance capture for computer graphics applications.
Impact The primary outputs were two publications at ACM SIGGRAPH, which already are considered milestone contributions by many: Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. Practical SVBRDF capture in the frequency domain. ACM Trans. on Graphics (Proc. SIGGRAPH), 32(4):110:1-110:12, 2013. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. (Proc. SIGGRAPH), 34(4):110:1-110:13, July 2015. In addition, our team is currently preparing a patent application (preliminary application already filed) on the two-shot SVBRDF capture approach. We anticipate economic impact soon.
Start Year 2010
 
Description Appearance Capture with Aalto and NVIDIA 
Organisation NVIDIA
Country Global 
Sector Private 
PI Contribution In joint research on practical appearance capture, I contributed with my general background in the field: within our team, I was the senior domain expert on appearance capture. Through weekly project meetings with Prof. Lehtinen (Aalto University, NVIDIA Research) and his graduate student Miika Aittala, we jointly pushed forward Mr. Aittala's research agenda, leading to publications at the top venue in our field. Later in the collaboration, there were additional synergies with my EPSRC-funded research on landscape appearance acquisition.
Collaborator Contribution Prof. Lehtinen, who has ample experience in industrial computer graphics research, specialising in rendering and natural light and reflectance phenomena, contributed both through his significant technical and scientific background, but also through the direct supervision of his graduate student Miika Aittala, who himself worked on material appearance capture for computer graphics applications.
Impact The primary outputs were two publications at ACM SIGGRAPH, which already are considered milestone contributions by many: Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. Practical SVBRDF capture in the frequency domain. ACM Trans. on Graphics (Proc. SIGGRAPH), 32(4):110:1-110:12, 2013. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. (Proc. SIGGRAPH), 34(4):110:1-110:13, July 2015. In addition, our team is currently preparing a patent application (preliminary application already filed) on the two-shot SVBRDF capture approach. We anticipate economic impact soon.
Start Year 2010
 
Description Depth-Camera-Based Scene Reconstruction with University of Siegen and MS Research 
Organisation University of Siegen
Country Germany 
Sector Academic/University 
PI Contribution My key contribution was as domain expert in the area of point-based computer graphics, with secondary contributions through my general experience in computational photography and 3D reconstruction. No other team member from my group at UCL contributed. The (still ongoing) collaboration has great synergies with my EPSRC-funded work on 3D reconstruction of landscapes.
Collaborator Contribution University of Siegen contributed through the work by the graduate student Damien Lefloch, under the supervision of Prof. Andreas Kolb, and more recently through additional support by their graduate students Hamed Sarbolandi and Makus Kluge.
Impact So far, one successful publication, and one journal submission currently undergoing minor revisions: [accepted] Damien Lefloch, Tim Weyrich, and Andreas Kolb. Anisotropic point-based fusion. In Proceedings of International Conference on Information Fusion (FUSION), pages 1-9. ISIF, July 2015. [undergoing minor revisions] Damien Lefloch, Hamed Sarbolandi, Tim Weyrich, Andreas Kolb. Comprehensive Use of Curvature For Robust And Accurate Online Surface Reconstruction. Conditionally accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI).
Start Year 2013
 
Description Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation British Broadcasting Corporation (BBC)
Department BBC Research & Development
Country United Kingdom 
Sector Public 
PI Contribution We provide core research capability and academic impacy.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Bath's doctoral training centre, the EPSRC funded Centre for Digital Entertainment is a viable option.
Start Year 2012
 
Description Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation Double Negative
Country United Kingdom 
Sector Private 
PI Contribution We provide core research capability and academic impacy.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Bath's doctoral training centre, the EPSRC funded Centre for Digital Entertainment is a viable option.
Start Year 2012
 
Description Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation Gobo Games
Country United States 
Sector Private 
PI Contribution We provide core research capability and academic impacy.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Bath's doctoral training centre, the EPSRC funded Centre for Digital Entertainment is a viable option.
Start Year 2012
 
Description Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation The Foundry Visionmongers Ltd
Country United Kingdom 
Sector Private 
PI Contribution We provide core research capability and academic impacy.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Bath's doctoral training centre, the EPSRC funded Centre for Digital Entertainment is a viable option.
Start Year 2012
 
Description OAK Academic Partners at Bath 
Organisation University of Bath
Department Department of Computer Science
Country United Kingdom 
Sector Academic/University 
PI Contribution We collaborate on research, each way. This includes UCL staff visiting Bath.
Collaborator Contribution We collaborate on research, each way. This includes Bath staff visiting UCL.
Impact We have held several meetings, including with the IAP. We have identified further leads, such as a collaboration with the British Library.
Start Year 2013
 
Description OAK Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation British Broadcasting Corporation (BBC)
Department BBC Research & Development
Country United Kingdom 
Sector Public 
PI Contribution We provide core research capability and academic impact.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Both UCL's PI and Co-I have practical experience with such transfer, through UCL Business and UCL Media Institute.
Start Year 2013
 
Description OAK Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation Double Negative
Country United Kingdom 
Sector Private 
PI Contribution We provide core research capability and academic impact.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Both UCL's PI and Co-I have practical experience with such transfer, through UCL Business and UCL Media Institute.
Start Year 2013
 
Description OAK Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation Gobo Games
Country United States 
Sector Private 
PI Contribution We provide core research capability and academic impact.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Both UCL's PI and Co-I have practical experience with such transfer, through UCL Business and UCL Media Institute.
Start Year 2013
 
Description OAK Industrial Advisory Panel (BBC, Gobo, DNeg, Foundry) 
Organisation The Foundry Visionmongers Ltd
Country United Kingdom 
Sector Private 
PI Contribution We provide core research capability and academic impact.
Collaborator Contribution Partners provide problem directions, and avenue for industrial impact.
Impact We are actively engaged with partners to transfer our research IP into their hands. Both UCL's PI and Co-I have practical experience with such transfer, through UCL Business and UCL Media Institute.
Start Year 2013
 
Description UCL academic partners 
Organisation University College London
Department Department of Computer Science
Country United Kingdom 
Sector Academic/University 
PI Contribution We collaborate on research, each way. This includes Bath staff visiting UCL.
Collaborator Contribution We collaborate on research, each way. This include UCL staff visiting Bath.
Impact We have held several meetings, including with the IAP. We have identified further leads, such as the British Library.
Start Year 2013
 
Title VIRTUAL SCENE GENERATION BASED ON IMAGERY 
Description BACKGROUND A virtual world is a simulated environment in which users may interact with virtual objects and locations of the virtual world. Each user may control a respective avatar through which the user may interact with other users' avatars in the virtual world. An avatar generally provides a graphical representation of an individual within the virtual world environment. Avatars are usually presented to other users as two or three-dimensional graphical representations that resembles a human individual. Frequently, virtual worlds allow multiple users to enter the virtual environment and interact with one another. Virtual worlds are said to provide an immersive environment, as they typically appear similar to the real world and objects tend to follow rules related to gravity, topography, locomotion, physics and kinematics. Of course, virtual worlds can suspend or alter these rules as well as provide other imaginative or fanciful environments. Users typically communicate with one another through their avatars using text messages sent between avatars, real-time voice communication, gestures displayed by avatars, symbols visible in the virtual world, and the like. Some virtual worlds are described as being persistent. A persistent world provides an immersive environment (e.g., a fantasy setting used as a setting for a role-playing game, or a virtual world complete with land, buildings, towns, and economies) that is generally always available and where events continue to occur regardless of the presence of a given avatar. Thus, unlike more conventional online games or multi-user environments, the virtual world continues to exist and plots and events continue to unfold as users enter (and exit) the virtual world. Virtual environments are presented as images on a display screen and some virtual environment may allow users to record events that occur within the virtual environment. SUMMARY Embodiments presented in this disclosure provide a computer-implemented method of scene generation. The method includes receiving an image depicting a scene and annotated by a sparse set of labels. The method also includes generating, based on the sparse set of labels, a dense set of labels annotating the image and a density map associated with the image. The method also includes generating a virtual scene based on the dense set of labels and the density map, where the virtual scene is output. Other embodiments presented in this disclosure provide a computer-readable medium for scene generation and containing a program which, when executed, performs an operation that includes receiving an image depicting a scene and annotated by a sparse set of labels. The operation also includes generating, based on the sparse set of labels, a dense set of labels annotating the image and a density map associated with the image. The operation also includes generating a virtual scene based on the dense set of labels and the density map, where the virtual scene is output. Still other embodiments presented in this disclosure provide a system for scene generation. The system includes one or more computer processors and a memory containing a program which, when executed by the one or more computer processors, is configured to perform an operation that includes receiving an image depicting a scene and annotated by a sparse set of labels. The operation also includes generating, based on the sparse set of labels, a dense set of labels annotating the image and a density map associated with the image. The operation also includes generating a virtual scene based on the dense set of labels and the density map, where the virtual scene is output. 
IP Reference US9262853 
Protection Patent granted
Year Protection Granted 2016
Licensed No
Impact None so far.
 
Title Code and framework for Roto++: Accelerating Professional Rotoscoping using Shape Manifolds 
Description This software implements our rotoscoping system, and makes it easy for uses to add their own shape-analysis and user-interaction functions for professional-grade masking of video clips. The software is a working implementation of our SIGGRAPH 2016 paper, "Roto++: Accelerating Professional Rotoscoping using Shape Manifolds". 
Type Of Technology Software 
Year Produced 2016 
Impact The software has been downloaded almost 2000 times, and the associated video has received over 2000 hits on Youtube. The partner company, The Foundry, is continuing the collaboration, and will be developing a product branch based on this technology. 
URL http://visual.cs.ucl.ac.uk/pubs/rotopp/
 
Title Code for HarmonNetworks 
Description Code implementing our HarmonicNetworks, which gives a neural network the ability to see rotated images as if they were not rotated, or to measure the amount of rotation present. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Resulted in a publication in the highly regarded conference IEEE CVPR (top ranked publication in Computer Science, according to Google Scholar): Harmonic Networks: Deep Translation and Rotation Equivariance, led by Daniel Worrall 
URL https://github.com/deworrall92/harmonicConvolutions
 
Title Code for Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions 
Description Code implementing our IUI 2017 paper (received Honorable Mention award), "Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions", led by Thanapong Intharah. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Resulted in a full paper published at IUI 2017, which received an award, and was a featured demo at the conference: Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions. 
URL http://visual.cs.ucl.ac.uk/pubs/HILC/
 
Title Code for Responsive Action-based Video Synthesis 
Description This software implements the prototype described in our CHI 2017 paper. It allows casual users to injest a video, and turn various elements in the video into loopable clips, that can be triggered by keyboard or through other interfaces. 
Type Of Technology Software 
Year Produced 2017 
Impact The software led to our CHI 2017 paper, led by Corneliu Ilisescu, "Responsive Action-based Video Synthesis". 
URL http://visual.cs.ucl.ac.uk/pubs/actionVideo/
 
Title Code for Unsupervised Monocular Depth Estimation with Left-Right Consistency 
Description This software and and the associated scripts allow reproduction of our new system, that converts a color image into a depth-image. The software also makes it easy to re-train the statistical model, using other binocular stereo pairs of images. All this software underpins our CVPR 2017 paper. 
Type Of Technology Software 
Year Produced 2017 
Impact The algorithms behind this software have now been filed in a patent application, and we are seeking commercialization opportunities. 
URL http://visual.cs.ucl.ac.uk/pubs/monoDepth/
 
Title Mean Shift Library; 02/2016 
Description Dr Tom SF Haines developed this Mean Shift library. Mean shift is usually associated, in computer vision at least, with the segmentation of an image. Whilst this library supports that scenario, it is far more general. Mean shift is a gradient ascent method for finding the modes of a kernel density estimate, so this library is as much a kernel density estimation library as it is a mode finder. It includes the usual kernel bandwidth estimation methods and also supports subspace constrained mean shift, which finds edges/manifolds in noisy data. Support goes far beyond the typical Gaussian and Uniform kernels: It has ten kernel types, as well as the ability to combine them, with different kernels on different parts of a feature vector. The kernels include directional distributions, so it supports density estimation over the position and orientation of an object, for instance. It also supports the multiplication of density estimates, which allows you to perform non-parametric belief propagation using mean shift objects as the messages between random variables. Also has methods to approximate values such as the entropy of a density estimate, and the KL divergence between two estimates. 
Type Of Technology Software 
Year Produced 2016 
Open Source License? Yes  
Impact At this point, we have no reliable information on the user basis of this library. 
URL http://reality.cs.ucl.ac.uk/projects/haines/haines16meanshift.html
 
Title Software and data for Scalable Inside-Out Image-Based Rendering 
Description Software and data associated with our Siggraph Asia 2016 paper "Scalable Inside-Out Image-Based Rendering", led by Peter Hedman. 
Type Of Technology Software 
Year Produced 2016 
Impact The software and algorithms will likely be folded into a startup, based at INRIA Sophia Anitpolis, with collaboration by the different academic and industry partners. 
URL http://visual.cs.ucl.ac.uk/pubs/insideout/
 
Description BBC Interview for Handwriting Synthesis project 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Rory Cellan-Jones, BBC Technology correspondent came and interviewed me and postdoc Dr Tom Haines, to showcase our research. The project allows us to scan someone's handwriting sample, and then to create new text in that person's handwriting. The report and BBC video from this interview were featured as the main Technology story on the BBC website. Our youtube video has been viewed over 52,000 times.
Year(s) Of Engagement Activity 2016
URL http://www.bbc.co.uk/news/technology-37046477
 
Description Conference Organisation 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact We ran (as paper chair in 2013, and 2015, general chair in 2014, and in 2016) the Computational Visual Media Production conference, aimed at bringing together academics and industry in the creative sector. It therefore fits with our aims for pathways to impact, enabling OAK researchers to engage directly with industry.
Year(s) Of Engagement Activity 2013,2014,2015,2016
URL http://www.cvmp-conference.org/2014-Papers
 
Description Invited talk at Friedrich-Alexander University Erlangen-Nürnberg, 14/4/2016 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Invited talk at post-graduate seminar of that university. The talk was on "Problem-Aware Digitisation of Cultural Heritage Artefacts" and showcased a number of EPSRC-funded innovations in digitisation.
Year(s) Of Engagement Activity 2016
 
Description Invited talk at GRIS Kolloquium, Technical University of Darmstadt, Germany, 21/4/2016 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Invited talk at post-graduate seminar of that university. The talk was on "Application-Driven Appearance Digitisation" and showcased a number of EPSRC-funded innovations in digitisation and fabrication.
Year(s) Of Engagement Activity 2016
URL http://www.gris.tu-darmstadt.de/home/news/archive/index.en.htm
 
Description Playfair Capital 2015 AI Summitt 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Probably about 300 people attended this event organized to showcase how AI and Machine Learning progress could potentially be explored by and for private industry. G. Brostow was an invited speaker for the panel, and the event was held at Bloomberg News headquarters in London.
Year(s) Of Engagement Activity 2015