Deep Learning for Free-Viewpoint Video in Sports and Immersive VR Experiences
Lead Research Organisation:
University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP
Abstract
Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
Organisations
People |
ORCID iD |
| Adrian Hilton (Primary Supervisor) | |
| LEWIS BRIDGEMAN (Student) |
Publications
Bridgeman L
(2019)
Full-body Performance Capture of Sports from Multi-view Video (Short Paper)
Bridgeman L
(2019)
Multi-Person 3D Pose Estimation and Tracking in Sports
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509772/1 | 30/09/2016 | 29/09/2021 | |||
| 1976240 | Studentship | EP/N509772/1 | 30/09/2017 | 29/06/2021 | LEWIS BRIDGEMAN |
| Description | We have identified that model-based reconstruction of people from multi-view video proves a lightweight, but realistic method of generating full-body reconstructions of people in constrained environments. We have produced FVV rendering results on single people in constrained studio environments that are able to be played back in virutal reality (VR) and augmented reality (AR). Model-based reconstruction is able to capture finer details (such as fingers, and facial details) where other reconstruction methods fail. However, there is still room for improvement in capturing details not present within the model, such as clothing. Our work in "Multi-person 3D Pose Estimation and Tracking in Sports" provides a stepping-stone to FVV of multiple people in sports scenes. This work focuses on sorting and tracking pose estimations of multiple people from multiple camera views in sports scenes; pose estimations are a critical component of the model-based reconstruction pipeline. This work provides a new method for: correcting errors in pose estimations using multi-view consensus; associating 2D pose estimations between camera viewpoints; and sorting associated poses between frames to generate tracked 3D skeletons. Our approach achieves a significant improvement in speed over the state-of-the-art. "Full-body Performance Capture of Sports from Multi-view Video" extends our previous work by using the sorted pose estimations in a model-based reconstruction of multiple people in sports environments. We demonstrate results for our method on a soccer dataset comprising over 20 subjects. These initial results show that model-based reconstruction has the potential to provide smooth, temporally consistent reconstructions of multiple people on challenging sports datasets. |
| Exploitation Route | Our intermediate work on multiple person 3D pose estimation could find applications in a range of fields: motion capture & animation; sports player analysis; or even gait analysis in healthcare. The extension of this work in multi-person reconstruction could prove useful in the creative industry: the method allows us to generate 4D reconstructions of real human performances from video cameras, which could help to save hours of animators' and digital artists' time. |
| Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Leisure Activities including Sports Recreation and Tourism |