SMARTLENS: Closing the gap between real and virtual filming through smart high end lens calibration
Lead Participant:
MO-SYS ENGINEERING LIMITED
Abstract
The SMARTLENS project leverages Mo-Sys' leadership in studio-quality virtual production and opens up broad new markets to affordable high-quality video production.
Video is ubiquitous, comprising 82% of all Internet traffic and displacing other forms of media in advertising, in education, news and beyond; attracting professionals from across the creative industries (£268 Bn) to explore and participate in new forms of content production.
'Virtual Production' technology allows filmmakers to combine images captured on camera with computer-generated elements. The SMARTLENS project creates a new automated method of essential lens calibration, allowing significant cost-saving (and carbon-saving), as production teams no longer need to travel to remote locations to capture difficult shots.
This is a collaborative project building on long-time collaboration with Professor Simon Julier (UCL/TMI), innovating novel AI-based automated lens calibration. All lenses distort the path of light in some way: changing the focal length changes the field of view of the lens, and straight lines in the real world do not necessarily look like straight lines in the image plane. Failure to account for this distortion means that the virtual graphics do not line up with the real world, and any notion of the graphics being anchored in the real world is lost. Conventional lens calibration techniques are slow and cumbersome, requiring the operator to manually measure targets and camera movements repeatedly.
SMARTLENS develops a set of algorithms and techniques which will automate the calibration approach. First, a suitable model of lens parameters will be chosen. Conventional computer vision models do not describe effects such as depth-dependent radial distortion for out of focus images. Therefore, we will chose a suitable model from the photogrammetry literature. Second, we will develop the back end optimisation techniques which will fit the parameters to suitable data. Third, we will investigate the use of various kinds of calibration targets, including points and lines to see which ones are most robust over the range of operating conditions. Finally, we will investigate suitable practices which give the best performance.
Mo-Sys is the largest virtual production technology provider today, with a substantial IP portfolio and a sophisticated high-end customer base (Sony, Warner Bros, Disney, Netflix, BBC, NHK, Fox, ESPN, Sky, CNN). Credits range from BBC's Match of the Day to films such as Gravity and Life of Pi. Through this project, this collaboration now continues to innovate and extend lens calibration and tracking systems,
Video is ubiquitous, comprising 82% of all Internet traffic and displacing other forms of media in advertising, in education, news and beyond; attracting professionals from across the creative industries (£268 Bn) to explore and participate in new forms of content production.
'Virtual Production' technology allows filmmakers to combine images captured on camera with computer-generated elements. The SMARTLENS project creates a new automated method of essential lens calibration, allowing significant cost-saving (and carbon-saving), as production teams no longer need to travel to remote locations to capture difficult shots.
This is a collaborative project building on long-time collaboration with Professor Simon Julier (UCL/TMI), innovating novel AI-based automated lens calibration. All lenses distort the path of light in some way: changing the focal length changes the field of view of the lens, and straight lines in the real world do not necessarily look like straight lines in the image plane. Failure to account for this distortion means that the virtual graphics do not line up with the real world, and any notion of the graphics being anchored in the real world is lost. Conventional lens calibration techniques are slow and cumbersome, requiring the operator to manually measure targets and camera movements repeatedly.
SMARTLENS develops a set of algorithms and techniques which will automate the calibration approach. First, a suitable model of lens parameters will be chosen. Conventional computer vision models do not describe effects such as depth-dependent radial distortion for out of focus images. Therefore, we will chose a suitable model from the photogrammetry literature. Second, we will develop the back end optimisation techniques which will fit the parameters to suitable data. Third, we will investigate the use of various kinds of calibration targets, including points and lines to see which ones are most robust over the range of operating conditions. Finally, we will investigate suitable practices which give the best performance.
Mo-Sys is the largest virtual production technology provider today, with a substantial IP portfolio and a sophisticated high-end customer base (Sony, Warner Bros, Disney, Netflix, BBC, NHK, Fox, ESPN, Sky, CNN). Credits range from BBC's Match of the Day to films such as Gravity and Life of Pi. Through this project, this collaboration now continues to innovate and extend lens calibration and tracking systems,
Lead Participant | Project Cost | Grant Offer |
|---|---|---|
| MO-SYS ENGINEERING LIMITED | £401,901 | £ 281,331 |
Participant |
||
| MEDIA RESEARCH PARTNERS LIMITED | £95,880 | £ 67,116 |
People |
ORCID iD |
| Oleks Skrypnyk (Project Manager) |