Scene Processing with Machine Learnable and Semantically Parametrized Scene Representations
Lead Research Organisation:
University of Cambridge
Department Name: Computer Science and Technology
Abstract
Digital representations of visual reality and imagination is an integral part of almost all scientific disciplines, industry, and society. Imaging techniques and computer graphics have successfully solved the problems of creating accurate projections, e.g. image and video, of the real world over the last decades. However, such projections are fundamentally limited representations of the underlying scenes and only allow for passive consumption such as viewing on a screen. Many applications from image editing to augmented/virtual reality instead require active 3D exploration, creation, and editing of scenes, for which we need full virtual scene models.
Creating virtual scenes that are high fidelity models of the real world and beyond via digital modelling or capture has traditionally been a privilege only available to corporations with educated artists and engineers, working with complex software and hardware tools for countless hours. These professionals produce and work with carefully designed parametrizations of geometry, appearance, and motion, which allow them to author and edit virtual scenes with all the intricate details of reality and their imagination. On the other end, many computer vision techniques try to have a shortcut by capturing scenes from the real world via simpler sensors in uncontrolled environments, or altering images of scenes with algorithms that only implicitly encode scene semantics, e.g. in latent spaces of artificial neural networks. These lead to scene representations of lower quality and in a form that is not easily editable for semantically meaningful modelling.
The objective of this research project is to tackle these shortcomings and develop a scene representation that is 1) a high fidelity detailed model of visual reality in terms of geometry, appearance, and motion, 2) machine learnable via capture in partially controlled practical environments, 3) semantically parametrized to allow for easy and intuitive edits, 4) fast to visualize for real-time exploration. Based on this representation, we will develop scene processing techniques that will allow individuals to create, alter, explore, and share high fidelity virtual objects and scenes, unlocking a completely new set of applications in augmented/virtual reality, gaming, product design, manufacturing, education, robotics, and medical domains.
Creating virtual scenes that are high fidelity models of the real world and beyond via digital modelling or capture has traditionally been a privilege only available to corporations with educated artists and engineers, working with complex software and hardware tools for countless hours. These professionals produce and work with carefully designed parametrizations of geometry, appearance, and motion, which allow them to author and edit virtual scenes with all the intricate details of reality and their imagination. On the other end, many computer vision techniques try to have a shortcut by capturing scenes from the real world via simpler sensors in uncontrolled environments, or altering images of scenes with algorithms that only implicitly encode scene semantics, e.g. in latent spaces of artificial neural networks. These lead to scene representations of lower quality and in a form that is not easily editable for semantically meaningful modelling.
The objective of this research project is to tackle these shortcomings and develop a scene representation that is 1) a high fidelity detailed model of visual reality in terms of geometry, appearance, and motion, 2) machine learnable via capture in partially controlled practical environments, 3) semantically parametrized to allow for easy and intuitive edits, 4) fast to visualize for real-time exploration. Based on this representation, we will develop scene processing techniques that will allow individuals to create, alter, explore, and share high fidelity virtual objects and scenes, unlocking a completely new set of applications in augmented/virtual reality, gaming, product design, manufacturing, education, robotics, and medical domains.
Planned Impact
This project is on digital modelling, capture, and visualization of 3D reality, which has become one of the main driving forces behind the progress in science, industry, and society. Its impact thus extends far beyond the primary scientific disciplines it rests on, computer graphics, computer vision, and machine learning.
The application areas include augmented/ virtual reality (AR/VR), video games, filmmaking, telecommunication, product design, manufacturing, health, robotics, education, and smart cities. In many of these areas, academia and industry are working closely together. Apart from scientific impact, the project thus has the potential to have a significant short and long-term impact on industry, which also extends to the public sector e.g. digital exhibitions and museums, improved healthcare services, or urban planning.
Digital capture and interactive visualizations of scenes have the potential to transform the way many industries are currently run. I have already led several research projects in the creative industries with the long-term goal of transforming the way video games and films are created via novel 3D capture tools, AR/VR displays, and human computer interfaces; and providing immersive 3D experiences for films, games, and telecommunications. Microsoft, Facebook, Apple, and Google are heavily investing in AR/VR devices for similar applications where virtual scenes are created, edited, and explored.
3D digital reality is also transforming product design, where designers can digitally create and optimize products without having to build a physical prototype, and do fast prototyping without going through expensive manufacturing cycles. Accurate digital models with precise overlays on reality via AR devices can then be used to guide manufacturing processes for reduced errors and costs. Such technologies are already attracting massive attention from many industries such as automotive and aviation. The outputs of this project will help to capture, track, and edit 3D objects for these applications.
Healthcare is another major area where 3D scene processing has numerous important applications. AR/VR based systems are being explored for therapy e.g. for mental problems, addictions, phobias, or problems associated with missing limbs. Further use cases are diagnosis by overlaying virtual layers of tissues or organs and showing relevant information, surgery planning, assistance, and remote operations with virtual models of human bodies.
Robotic systems fundamentally rely on building digital models of environments to navigate and perform tasks. All robotics applications will thus fundamentally benefit from the developed scene representations and techniques. High fidelity scene representations can transform human robot interactions by accurate perception of humans and robot-human coordination. Such systems have a diversity of applications such as rescue operations, law enforcement, assisting disabled individuals, or creating artworks.
Editable and high fidelity digital models of reality capturing objects from furniture or buildings to entire cities or nature can also be instrumental for education and smart cities with immersive 3D visualizations for training, AR/VR systems for individuals with disabilities, preserving and visualizing cultural heritage, urban planning, or navigation.
The application areas include augmented/ virtual reality (AR/VR), video games, filmmaking, telecommunication, product design, manufacturing, health, robotics, education, and smart cities. In many of these areas, academia and industry are working closely together. Apart from scientific impact, the project thus has the potential to have a significant short and long-term impact on industry, which also extends to the public sector e.g. digital exhibitions and museums, improved healthcare services, or urban planning.
Digital capture and interactive visualizations of scenes have the potential to transform the way many industries are currently run. I have already led several research projects in the creative industries with the long-term goal of transforming the way video games and films are created via novel 3D capture tools, AR/VR displays, and human computer interfaces; and providing immersive 3D experiences for films, games, and telecommunications. Microsoft, Facebook, Apple, and Google are heavily investing in AR/VR devices for similar applications where virtual scenes are created, edited, and explored.
3D digital reality is also transforming product design, where designers can digitally create and optimize products without having to build a physical prototype, and do fast prototyping without going through expensive manufacturing cycles. Accurate digital models with precise overlays on reality via AR devices can then be used to guide manufacturing processes for reduced errors and costs. Such technologies are already attracting massive attention from many industries such as automotive and aviation. The outputs of this project will help to capture, track, and edit 3D objects for these applications.
Healthcare is another major area where 3D scene processing has numerous important applications. AR/VR based systems are being explored for therapy e.g. for mental problems, addictions, phobias, or problems associated with missing limbs. Further use cases are diagnosis by overlaying virtual layers of tissues or organs and showing relevant information, surgery planning, assistance, and remote operations with virtual models of human bodies.
Robotic systems fundamentally rely on building digital models of environments to navigate and perform tasks. All robotics applications will thus fundamentally benefit from the developed scene representations and techniques. High fidelity scene representations can transform human robot interactions by accurate perception of humans and robot-human coordination. Such systems have a diversity of applications such as rescue operations, law enforcement, assisting disabled individuals, or creating artworks.
Editable and high fidelity digital models of reality capturing objects from furniture or buildings to entire cities or nature can also be instrumental for education and smart cities with immersive 3D visualizations for training, AR/VR systems for individuals with disabilities, preserving and visualizing cultural heritage, urban planning, or navigation.
Organisations
- University of Cambridge (Fellow, Lead Research Organisation)
- University College London (Collaboration)
- HARVARD UNIVERSITY (Collaboration)
- ETH Zurich (Collaboration)
- McGill University (Collaboration)
- Purdue University (Collaboration)
- Institute of Information Science and Technologies (Collaboration)
- Adobe Inc. (Collaboration)
- Disney Research Zurich (Collaboration)
- ServiceNow (Collaboration)
- DEEPMIND TECHNOLOGIES LIMITED (Collaboration)
- Massachusetts Institute of Technology (Collaboration)
- University of Lincoln (Collaboration)
- Max Planck Society (Collaboration)
- Google (Collaboration)
- Simon Fraser University (Collaboration)
- Unity Technologies (Collaboration)
- The Hong Kong University of Science and Technology (Collaboration)
- University of California, San Diego (UCSD) (Collaboration)
- Mila - Quebec AI Institute (Collaboration)
- University of Adelaide (Collaboration)
- University of Toronto (Collaboration)
- UNIVERSITY OF BRITISH COLUMBIA (Collaboration)
People |
ORCID iD |
Ahmet Oztireli (Principal Investigator / Fellow) |
Publications
Chenliang Zhou
(2023)
FrePolad: Frequency-Rectified Point Latent Diffusion for Point Cloud Generation
Dodik A
(2021)
Path Guiding Using Spatio-Directional Mixture Models
in Computer Graphics Forum
Fangcheng Zhong
(2023)
Neural Fields with Hard Constraints of Arbitrary Differential Order
Fazilet Gokbudak
(2023)
Hypernetworks for Generalizable BRDF Representation
Fei Yin
(2023)
3d gan inversion with facial symmetry prior
Greff K
(2022)
Kubric: A scalable dataset generator
Jack Foster
(2024)
Zero-Shot Machine Unlearning at Scale via Lipschitz Regularization
Description | The research programme funded by my Future Leaders Fellowship is on solving one of the grand challenges of science and engineering: creating digital models of visual reality and human imagination. I have been working on the highly ambitious goal of delivering high-fidelity, machine-learnable, and human-controllable digital scene representations and associated scene processing algorithms. These scene representations will lead to a fundamental change in the way we create, edit, and visualise 3D data. This powerful technology has the potential to revolutionise not only my field but also the scientific disciplines, industries, and public services that rely on digital 3D data and digital twins. Scene representations underlie all such algorithms and systems we use in practice. |
Exploitation Route | Digital 3D capture and synthesis are essential building blocks for any applications utilizing digital twins or simulations. |
Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Environment Healthcare Manufacturing including Industrial Biotechology Culture Heritage Museums and Collections Retail |
Description | We have developed two open source large codebases with the involvement of many partners from academia and industry. These have already been utilised for many applications in industry. |
First Year Of Impact | 2021 |
Sector | Digital/Communication/Information Technologies (including Software) |
Impact Types | Economic |
Description | New Course on Extended Reality at the University of Cambridge |
Geographic Reach | Local/Municipal/Regional |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | This course is the first of its kind at the University of Cambridge to our knowledge. It provided the essential skills to build extended reality applications. These will be key skills in the digital industry. |
Description | New Course on Machine Visual Perception at the University of Cambridge |
Geographic Reach | Europe |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | This course allowed the graduate students to understand the details of several techniques heavily utilized in academia and industry in the fields of machine learning, computer vision, and graphics. They had practical experience via a course project and practicals. |
Description | AgriFoRwArdS CDT |
Amount | £150,000 (GBP) |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2022 |
End | 10/2025 |
Description | Qualcomm Studentship |
Amount | £150,000 (GBP) |
Organisation | Qualcomm |
Sector | Private |
Country | United States |
Start | 09/2023 |
End | 10/2026 |
Title | Kubric: A scalable dataset generator |
Description | Kubric: A scalable dataset generator Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing, and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness, and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to the fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2022 |
Provided To Others? | Yes |
Impact | The publication is accepted to CVPR 2022. The codebase is publically available for research. |
Title | Kubric: A scalable dataset generator |
Description | Kubric: A scalable dataset generator Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing, and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness, and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to the fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification. |
Type Of Material | Data handling & control |
Year Produced | 2022 |
Provided To Others? | Yes |
Impact | The publication associated with this technique is accepted to CVPR 2022 and the code is publically available for research. |
Title | Multi-view video datasets for evaluation of NeRF and Neural View Synthesis methods |
Description | The dataset contains video frames and camera matrices intended for testing NeRF and Neural View Synthesis methods. The frames were captured with Sony A7RIII camera, either in a controlled laboratory setup on the streets in the Natural History Museum of the University of Pisa The scenes are mostly static and the motion is due to camera movement. |
Type Of Material | Database/Collection of data |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | This is the first comprehensive dataset to evaluate modern view synthesis techniques. |
URL | https://www.repository.cam.ac.uk/items/6a720222-163a-45bd-8442-198a36ad4290 |
Description | Research collaboration |
Organisation | Adobe Inc. |
Country | United States |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | DeepMind Technologies Limited |
Country | United Kingdom |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Disney Research Zurich |
Country | Switzerland |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | ETH Zurich |
Country | Switzerland |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | |
Country | United States |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Harvard University |
Country | United States |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Institute of Information Science and Technologies |
Country | Italy |
Sector | Public |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Massachusetts Institute of Technology |
Country | United States |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Max Planck Society |
Department | Max Planck Institute for Informatics |
Country | Germany |
Sector | Charity/Non Profit |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | McGill University |
Country | Canada |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Mila - Quebec AI Institute |
Country | Canada |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Purdue University |
Country | United States |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Purdue University |
Country | United States |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | ServiceNow |
Country | United States |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Simon Fraser University |
Country | Canada |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | The Hong Kong University of Science and Technology |
Country | Hong Kong |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | Unity Technologies |
Country | United States |
Sector | Private |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University College London |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University of Adelaide |
Country | Australia |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University of Adelaide |
Country | Australia |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University of British Columbia |
Country | Canada |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University of California, San Diego (UCSD) |
Country | United States |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University of Lincoln |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Description | Research collaboration |
Organisation | University of Toronto |
Country | Canada |
Sector | Academic/University |
PI Contribution | We contribute with our research expertise, developing ideas and computational frameworks, and computational experiments. |
Collaborator Contribution | The researchers from the partner organizations are similarly contributing with their research expertise, computational frameworks, experiments, and donations. |
Impact | Publication: Blue noise for diffusion models (arXiv 2024) Publication: Dream: Visual decoding from reversing human visual system (WACV 2024) Publication: Differentiable visual computing for inverse problems and machine learning (Nature Machine Intelligence 2023) Publication: Neural fields with hard constraints of arbitrary differential order (NeurIPS 2023) Publication: Statistical shape representations for temporal registration of plant components in 3D (ICRA 2023) Publication: 3d gan inversion with facial symmetry prior (CVPR 2023) Publication: Controllable shadow generation using pixel height maps (ECCV 2022) Publication: Kubric: A scalable dataset generator (CVPR 2022) Publication: Controllable Shadow Generation Using Pixel Height Maps (ECCV 2022) Publication: D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video (NeurIPS 2022) Publication: Instance segmentation, body part parsing, and pose estimation of human figures in pictorial maps (International Journal of Cartography 8 (3), 291-307) Publication: Path Guiding Using Spatio-Directional Mixture Models (Computer Graphics Forum 41 (1), 172-189) Publication: Iso-points: Optimizing neural implicit surfaces with hybrid representations (CVPR 2021) |
Start Year | 2021 |
Title | Constrained Neural Fields |
Description | This repository is the official implementation of the paper Neural Fields with Hard Constraints of Arbitrary Differential Order, NeurIPS 2023. |
Type Of Technology | Software |
Year Produced | 2023 |
Open Source License? | Yes |
Impact | This is the first general method for exact constraints for neural fields. |
URL | https://github.com/zfc946/CNF |
Title | D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video |
Description | This is the codebase for "D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video". |
Type Of Technology | Software |
Year Produced | 2022 |
Open Source License? | Yes |
Impact | N/A |
URL | https://github.com/ChikaYan/d2nerf |
Title | Kubric: A scalable dataset generator |
Description | Kubric: A scalable dataset generator Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing, and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness, and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to the fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification. |
Type Of Technology | Software |
Year Produced | 2022 |
Open Source License? | Yes |
Impact | - |
URL | https://github.com/google-research/kubric |
Description | Invited Speaker at Koc University, Turkey |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | Invited Speaker at Koc University, Turkey Title: 3D Digital Reality Modeling for Perception Abstract: Creating digital models of reality is one of the grand challenges of computer science. In this talk, I will summarize some of our efforts towards achieving this goal to allow machines to perceive the world as well as and beyond humans. The focus will be on capturing and replicating the visual world and techniques at the intersection of computer graphics, vision, and machine learning to solve several fundamental problems and their practical applications. |
Year(s) Of Engagement Activity | 2021 |
Description | Invited Speaker at OSA Incubator on Perception in Immersive Technologies |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Invited Speaker at OSA Incubator on Perception in Immersive Technologies Topic: 3D Scene Representations |
Year(s) Of Engagement Activity | 2021 |
Description | Invited Speaker at Sutton Trust Summer School |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Undergraduate students |
Results and Impact | Invited Speaker at Sutton Trust Summer School Topic: Creating Artwork Digitally |
Year(s) Of Engagement Activity | 2021 |
Description | Invited Speaker at University College London |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | Invited Speaker at University College London Title: 3D Digital Reality - Modeling for Perception Abstract: Creating digital models of reality is one of the grand challenges of computer science. In this talk, I will summarize some of our efforts towards achieving this goal to allow machines to perceive the world as well as and beyond humans. The focus will be on capturing and replicating the visual world and techniques at the intersection of computer graphics, vision, and machine learning to solve several fundamental problems and their practical applications. |
Year(s) Of Engagement Activity | 2021 |
Description | Invited Talk at Making Visual Art/Work in the AI Era |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | Invited talk at the event Making Visual Art/Work in the AI Era on generative AI. |
Year(s) Of Engagement Activity | 2023 |
Description | Keynote Speaker at CVMP 2021 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Keynote Speaker at CVMP 2021 Title: 3D Digital Reality -- Modeling for Perception Abstract: Creating digital models of reality is one of the grand challenges of computer science. In this talk, I will summarize some of our efforts towards achieving this goal to allow machines to perceive the world as well as and beyond humans. The focus will be on capturing and replicating the visual world and techniques at the intersection of computer graphics, vision, and machine learning to solve several fundamental problems and their practical applications. |
Year(s) Of Engagement Activity | 2021 |
Description | Keynote Speaker at ICCV Diff. 3D Vision and Graphics Workshop |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Keynote Speaker Title: 3D Digital Reality -- Modeling for Perception Abstract: Creating digital models of reality is one of the grand challenges of computer science. In this talk, I will summarize some of our efforts towards achieving this goal to allow machines to perceive the world as well as and beyond humans. The focus will be on capturing and replicating the visual world and techniques at the intersection of computer graphics, vision, and machine learning to solve several fundamental problems and their practical applications. |
Year(s) Of Engagement Activity | 2021 |
Description | Keynote Speaker at Sabanci University, Turkey |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | Keynote Speaker at Sabanci University, Turkey Title: Interpretable Machine Learning |
Year(s) Of Engagement Activity | 2021 |
Description | Keynote Speaker at Sabanci University, Turkey |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | Keynote Speaker at Sabanci University, Turkey Title: 3D Digital Reality -- Modeling for Perception Abstract: Creating digital models of reality is one of the grand challenges of computer science. In this talk, I will summarize some of our efforts towards achieving this goal to allow machines to perceive the world as well as and beyond humans. The focus will be on capturing and replicating the visual world and techniques at the intersection of computer graphics, vision, and machine learning to solve several fundamental problems and their practical applications. |
Year(s) Of Engagement Activity | 2021 |