Open3D: Collaborative Editing for 3D Virtual Worlds
Lead Research Organisation:
University College London
Department Name: Computer Science
Abstract
Digital 3D models are used in almost all areas of design and engineering from drug discovery through to architectural design optimization. With the advent of new imaging technologies such as more advanced remote sensing systems and consumer 3D cameras, there is a new capability to capture models that are both deep and broad in detail.
Currently there is a range of different software packages for editing 3D data but each is rather specialised to its domain. If data is to be exchanged between different users then the most usual way of doing this is to export files from one machine, move the file and import it on another machine. This can be facilitated by various file-sharing systems, but the unit of access is a file on a local file system. This presents multiple problems: as models grow in complexity managing them in files becomes problematic, collaborative editing is very hard and tracking of changes becomes challenging. In addition in the next few years, we should expect 3D model sizes to grow in size and detail at increasing rates. This will be driven both by consumer editing and scanning tools, but also increasing use of commercially scanned and produced models. One can extrapolate from the current extensive but crude representations of cities on Google Earth, or the highly detailed, but ultimately limited in scale models in modern video games, to imagine that models will reach 10^9 - 10^11+ polygons in scale within the next decade.
There is a domain where collaborative access to large models has been solved: Internet storage of documents. Systems as diverse as Wikipedia and Google Docs demonstrate that by decoupling storage from viewing and editing, extremely large repositories of information can be built.
In Open3D we will design the necessary algorithms and services that allow the hosting of 3D models of such scale on the Internet. Such 3D models, however, are fundamentally different from text counterparts: 3D models extend across space (e.g. 2-manifold data), lack an obvious extrinsic parameterization, and can have large variations in local details; while, text documents have a natural linear ordering making them much simpler to work with. Our observation is that while models may be big, the spatial scope of an editing or visualisation is usually limited. Thus we can imagine a network protocol that can exchange model assets based on spatial queries, rather than file access. Further, we can imagine that we can perform locking and revision control on this model to prevent inconsistent model states across multiple editors.
Through Open3D's unique set of facilities we want to enable synchronous collaborative modelling of a unified model with the minimum of interference in the user experience. By doing this we hope to enable a new crowd-sourcing effort to develop large-scale models in a couple of domains. In particular, in our impact plan we target creating a virtual model of part of London, and an open access anatomical model of the human body.
Currently there is a range of different software packages for editing 3D data but each is rather specialised to its domain. If data is to be exchanged between different users then the most usual way of doing this is to export files from one machine, move the file and import it on another machine. This can be facilitated by various file-sharing systems, but the unit of access is a file on a local file system. This presents multiple problems: as models grow in complexity managing them in files becomes problematic, collaborative editing is very hard and tracking of changes becomes challenging. In addition in the next few years, we should expect 3D model sizes to grow in size and detail at increasing rates. This will be driven both by consumer editing and scanning tools, but also increasing use of commercially scanned and produced models. One can extrapolate from the current extensive but crude representations of cities on Google Earth, or the highly detailed, but ultimately limited in scale models in modern video games, to imagine that models will reach 10^9 - 10^11+ polygons in scale within the next decade.
There is a domain where collaborative access to large models has been solved: Internet storage of documents. Systems as diverse as Wikipedia and Google Docs demonstrate that by decoupling storage from viewing and editing, extremely large repositories of information can be built.
In Open3D we will design the necessary algorithms and services that allow the hosting of 3D models of such scale on the Internet. Such 3D models, however, are fundamentally different from text counterparts: 3D models extend across space (e.g. 2-manifold data), lack an obvious extrinsic parameterization, and can have large variations in local details; while, text documents have a natural linear ordering making them much simpler to work with. Our observation is that while models may be big, the spatial scope of an editing or visualisation is usually limited. Thus we can imagine a network protocol that can exchange model assets based on spatial queries, rather than file access. Further, we can imagine that we can perform locking and revision control on this model to prevent inconsistent model states across multiple editors.
Through Open3D's unique set of facilities we want to enable synchronous collaborative modelling of a unified model with the minimum of interference in the user experience. By doing this we hope to enable a new crowd-sourcing effort to develop large-scale models in a couple of domains. In particular, in our impact plan we target creating a virtual model of part of London, and an open access anatomical model of the human body.
Planned Impact
This project will have impact across a wide range of areas. 3D models are becoming pervasive in many domains. Access to shared 3D models as easily as other types of documents on the web will be disruptive. It will definitely cut costs of access to shared models, but also enable new types of working. If we can kick-start a new type of crowd-sourcing effort, the potential impact could be very significant.
One example area of long-term impact will be, architecture, civil engineering and building services. The UK Government recently started to require the use of 3D formats (Building Information Models or BIM) in all centrally procured projects. From the client (government) point of view, the introduction of BIM will lead to efficiency, both in energy and cost, in the whole life cycle of buildings. The potential cost savings in building and other infrastructure management could be vast, and Open3D has a potential role to play in starting a service community around virtual models that can reliably predict the utilization of space and the resources needed for it. To set a 50-year vision, what if the built environment was entirely modelled in 3D down to individual building elements. This could have a very wide range of applications for smart building and infrastructure projects, heritage, and recreation. A detailed 3D model of a space, which could be updated based on manual and autonomous sensing, would enable a potential future where autonomous and semi-autonomous robots are fulfilling more complex transport and service roles. It would also be necessary for certain forms of augmented reality, where a 3D model is needed to control the rendering and visibility of objects. The same technology can be equally applied to Computer Aided Design (CAD) modelling for aerospace and automotive industries where much of the design and prototyping is performed digitally.
Another example of long-term impact can be found in the games industry. Part of the inspiration for work leading to this proposal came from one of the PI's own experience at Electronic Arts on constructing large models. Another UK studio, Rockstar North, with Grand Theft Auto V has created a large game world for players. The UK needs to keep at the forefront in tools for games development, so that its competitive advantage in this field is maintained. In the short to mid-term, we can envisage new tools improving the efficiency of games studios, and enabling medium-sized studios to develop larger games. To give a slightly less speculative vision, say 10-15 rather than 50 years, what if we could have fully open editable worlds in which multiple parties could edit and set tasks. This is partly the vision of various science-fiction/cyberpunk metaverse or cyberspace environments. The only large-scale systems of this nature are either completely closed (e.g., World of Warcraft), or are partly open, but still commercial (e.g., SecondLife). We can envisage an architecture where the editing and construction of worlds is decoupled from game-playing mechanics, so that multiple providers of content can utilize the under-lying virtual world, building independent or cooperating systems that use the same space. Imagine a version of Gotham City, Middle-Earth, or Rama that anyone could build games for.
To secure impact stories such as these we will undertake two case study demonstrators in this project: one on a virtual city model of London and another on anatomical modelling. These demonstrators were chosen because of the different types of impact: the first being mostly in business leading to greater efficiency, the second being within the university teaching domain and thus potentially contributing to medical training and research. We will also run workshops to publicize the open tools, with an eye on engaging with computer games and media industries in London and at a national level.
One example area of long-term impact will be, architecture, civil engineering and building services. The UK Government recently started to require the use of 3D formats (Building Information Models or BIM) in all centrally procured projects. From the client (government) point of view, the introduction of BIM will lead to efficiency, both in energy and cost, in the whole life cycle of buildings. The potential cost savings in building and other infrastructure management could be vast, and Open3D has a potential role to play in starting a service community around virtual models that can reliably predict the utilization of space and the resources needed for it. To set a 50-year vision, what if the built environment was entirely modelled in 3D down to individual building elements. This could have a very wide range of applications for smart building and infrastructure projects, heritage, and recreation. A detailed 3D model of a space, which could be updated based on manual and autonomous sensing, would enable a potential future where autonomous and semi-autonomous robots are fulfilling more complex transport and service roles. It would also be necessary for certain forms of augmented reality, where a 3D model is needed to control the rendering and visibility of objects. The same technology can be equally applied to Computer Aided Design (CAD) modelling for aerospace and automotive industries where much of the design and prototyping is performed digitally.
Another example of long-term impact can be found in the games industry. Part of the inspiration for work leading to this proposal came from one of the PI's own experience at Electronic Arts on constructing large models. Another UK studio, Rockstar North, with Grand Theft Auto V has created a large game world for players. The UK needs to keep at the forefront in tools for games development, so that its competitive advantage in this field is maintained. In the short to mid-term, we can envisage new tools improving the efficiency of games studios, and enabling medium-sized studios to develop larger games. To give a slightly less speculative vision, say 10-15 rather than 50 years, what if we could have fully open editable worlds in which multiple parties could edit and set tasks. This is partly the vision of various science-fiction/cyberpunk metaverse or cyberspace environments. The only large-scale systems of this nature are either completely closed (e.g., World of Warcraft), or are partly open, but still commercial (e.g., SecondLife). We can envisage an architecture where the editing and construction of worlds is decoupled from game-playing mechanics, so that multiple providers of content can utilize the under-lying virtual world, building independent or cooperating systems that use the same space. Imagine a version of Gotham City, Middle-Earth, or Rama that anyone could build games for.
To secure impact stories such as these we will undertake two case study demonstrators in this project: one on a virtual city model of London and another on anatomical modelling. These demonstrators were chosen because of the different types of impact: the first being mostly in business leading to greater efficiency, the second being within the university teaching domain and thus potentially contributing to medical training and research. We will also run workshops to publicize the open tools, with an eye on engaging with computer games and media industries in London and at a national level.
Publications
Guerrero P
(2016)
RAID a relation-augmented image descriptor
in ACM Transactions on Graphics
Guerrero P
(2018)
DepthCut: improved depth edge estimation using multiple unreliable channels
in The Visual Computer
Guerrero P
(2016)
PATEX exploring pattern variations
in ACM Transactions on Graphics
Jennett C
(2017)
Case Study 3 Students' Experiences of Interdisciplinary Learning while Building Scientific Video Games
in International Journal of Game-Based Learning
Jennett C
(2017)
Bored during the Christmas break? Get creative with science
Jennett C
(2016)
Creativity in Citizen Cyberscience
in Human Computation
Jennett C
(2017)
Designing for Curiosity in Citizen Science
Jennett C
(2017)
The Wiley Handbook of Human-Computer Interaction, Volume 2
Kamel Boulos MN
(2017)
From urban planning and emergency training to Pokémon Go: applications of virtual reality GIS (VRGIS) and augmented reality GIS (ARGIS) in personal, public and environmental health.
in International journal of health geographics
Kelly T
(2018)
FrankenGAN guided detail synthesis for building mass models using style-synchonized GANs
in ACM Transactions on Graphics
Lu Z
(2016)
Open3D
Lv Z
(2017)
Next-Generation Big Data Analytics: State of the Art, Challenges, and Future Research Topics
in IEEE Transactions on Industrial Informatics
Mitra N
(2019)
CreativeAI
Scully T
(2016)
glTF streaming from 3D repo to X3DOM
Zhao X
(2016)
Relationship templates for creating scene variations
in ACM Transactions on Graphics
Description | That is is possible to crowd-source complex 3D models. 3D model editing is a complex skill, so to enable broad access to crowd-sourced models, we have developed simplified and assisted model editing process to allow casual users to interact with the system. We have defined a new class of implicit parametric modelling systems where we can learn how to simplify parametric models. |
Exploitation Route | We are looking at developing new collaborations around new use cases. Our tools are agnostic to specifically what the application is. We have targeted city modelling for our own demonstrators, but users can appropriate the tools for completely new use cases. There is a developing use case, around using crowd-sourced 3D models for tracking for next-generation "visual GPS" services. We have started investigating that. There are good applications in immersive technologies, including the creation of new location-based systems based on the city. We have now undertaken a follow-up project within the CYENS Centre of Nicosia, blening our work with digital twins/smart cities work. |
Sectors | Construction Creative Economy Digital/Communication/Information Technologies (including Software) Education Environment Leisure Activities including Sports Recreation and Tourism Culture Heritage Museums and Collections Transport |
Description | We are starting to crowd source a city model of London. This can be compared to other existing surveyed models. It could be very high impact in the long run. We were able to demonstrate a process to elucidate complex, visually appealing models, from partial surveyed or crowd-sourced data. |
Sector | Construction,Creative Economy |
Impact Types | Cultural Societal |
Description | H2020-WIDESPREAD-01-2016-2017 |
Amount | € 15,000,000 (EUR) |
Funding ID | 739578 |
Organisation | European Commission H2020 |
Sector | Public |
Country | Belgium |
Start | 11/2017 |
End | 10/2024 |
Title | Ubiq |
Description | An online tool for teaching and research on social virtual reality systems |
Type Of Technology | Software |
Year Produced | 2021 |
Open Source License? | Yes |
Impact | This has been adopted by a number of universities for teaching, and has an increasing user base. |
URL | http://ubiq.online |
Description | "Science:Disrupt" podcast as part of their Citizen Science mini-series |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Dr Charlene Jennett was invited to take part in a "Science:Disrupt" podcast in July 2017 as part of their mini-series on citizen science. She was interviewed about her research investigating citizens' motivations for taking part in online citizen science projects and their experiences of learning and creativity. The editing of the podcast series has been delayed, but it is currently being expected to air in January/February 2018. Once the podcast is online, it could potentially reach an audience of 100+. The exact audience who listens to "Science:Disrupt" is unknown, but potentially this could be an international audience. |
Year(s) Of Engagement Activity | 2017 |
URL | https://sciencedisrupt.com/science-disruptors/collaboration-citizen-science |
Description | Ada Lovelace Day - Speed Networking with 14-16 year olds |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Schools |
Results and Impact | UCL Ada Lovelace Day was an outreach event run by UCL Computer Science on Tuesday 13th October 2015. The goal was to inspire 14-16 year old female students to think about STEM careers and to apply for STEM subjects at university. Roughly 70 students attended, accompanied by 6 teachers. During the event, students found out more about Ada Lovelace - the first ever computer programmer. Students also took part in a "speed networking" session, where they sat in small groups and had the chance to talk to UCL PhD students and Postdocs about their research. Dr Charlene Jennett was one of the researchers who took part in the speed-networking session and talked to students about her research and the field of Human-Computer Interaction. |
Year(s) Of Engagement Activity | 2015 |
URL | https://www.engineering.ucl.ac.uk/news/ucl-celebrates-ada-lovelace-day/ |
Description | in2scienceUK - STEM placement for 16-18 year olds |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Schools |
Results and Impact | in2scienceUK is an award-winning charity which empower students from disadvantaged backgrounds to achieve their potential and progress to STEM and research careers through high quality work placements and careers guidance. During the Summer of 2016, Dr Charlene Jennett volunteered as a STEM placement supervisor and organised work experience placements for two A-level students. Charlene talked to the students about her research and the field of Human-Computer Interaction. She also organised a 2-week schedule of activities, liasing with academic researchers from various departments within UCL, and former UCL students who were now working in industry. This enabled the students to learn about various kinds of Psychology and Computer Science research and careers, and hopefully inspired them to apply to university in the future. Charlene also helped the students to write daily blog posts about their activities, which she shared with the research departments involved, and which were further shared on social media. Here's a link to their in2science blog: https://uclicin2science.wordpress.com/ |
Year(s) Of Engagement Activity | 2016 |
URL | https://uclicin2science.wordpress.com/ |