COMPPACT: Compression of Video using Perceptually Optimised Parametric Coding Techniques
Lead Research Organisation:
University of Bristol
Department Name: Electrical and Electronic Engineering
Abstract
It is currently a very exciting and challenging time for video compression. The predicted growth in demand for bandwidth, especially for mobile services is driven largely by video applications and is probably greater now than it has ever been. There are four reasons for this:
(i) Recently introduced formats such as 3D and multiview, coupled with increasing dynamic range, spatial resolution and framerate, all require increased bit-rate to deliver improved immersion;
(ii) Video-based web traffic continues to grow and dominate the internet;
(iii) User expectations coninue to drive flexibility and quality, with a move from linear to non-linear delivery;
(iv) Finally the emergence of new services, in particular mobile delivery through 4G/LTE to smart phones. While advances in network and physical layer technologies will no doubt contribute to the solution, the role of video compression is also of key importance.
This research project is underpinned by the assumption that, in most cases, the target of video compression is to provide good subjective quality rather than to minimise the error between the original and coded pictures. It is thus possible to conceive of a compression scheme where an analysis/synthesis framework replaces the conventional energy minimisation approach. Such a scheme could offer substantially lower bitrates through reduced residual and motion vector coding.
The approach proposed will model scene content using combinations of waveform coding and texture replacement, using computer graphic models to replace target textures at the decoder. These not only offer the potential for dramatic improvements in performance, but they also provide an inherent content-related parameterisation which will be of use in classification and detection tasks as well as facilitating integration with CGI.
This has the potential to create a new content-driven framework for video compression. In this context our aim is to shift the video coding paradigm from rate-distortion optimisation to rate-quality modelling, where region-based parameters are combined with perceptual quality metrics to inform and drive the coding and synthesis processes. However it is clear that a huge amount of research needs to be done in order to fully exploit the method's potential and to yield stable and efficient solutions. For example, mean square error is no longer a valid objective function or measure of quality, and new embedded perceptually driven quality metrics are essential. The choice of texture analysis and synthesis models are also important, as is the exploitation of long-term picture dependencies.
(i) Recently introduced formats such as 3D and multiview, coupled with increasing dynamic range, spatial resolution and framerate, all require increased bit-rate to deliver improved immersion;
(ii) Video-based web traffic continues to grow and dominate the internet;
(iii) User expectations coninue to drive flexibility and quality, with a move from linear to non-linear delivery;
(iv) Finally the emergence of new services, in particular mobile delivery through 4G/LTE to smart phones. While advances in network and physical layer technologies will no doubt contribute to the solution, the role of video compression is also of key importance.
This research project is underpinned by the assumption that, in most cases, the target of video compression is to provide good subjective quality rather than to minimise the error between the original and coded pictures. It is thus possible to conceive of a compression scheme where an analysis/synthesis framework replaces the conventional energy minimisation approach. Such a scheme could offer substantially lower bitrates through reduced residual and motion vector coding.
The approach proposed will model scene content using combinations of waveform coding and texture replacement, using computer graphic models to replace target textures at the decoder. These not only offer the potential for dramatic improvements in performance, but they also provide an inherent content-related parameterisation which will be of use in classification and detection tasks as well as facilitating integration with CGI.
This has the potential to create a new content-driven framework for video compression. In this context our aim is to shift the video coding paradigm from rate-distortion optimisation to rate-quality modelling, where region-based parameters are combined with perceptual quality metrics to inform and drive the coding and synthesis processes. However it is clear that a huge amount of research needs to be done in order to fully exploit the method's potential and to yield stable and efficient solutions. For example, mean square error is no longer a valid objective function or measure of quality, and new embedded perceptually driven quality metrics are essential. The choice of texture analysis and synthesis models are also important, as is the exploitation of long-term picture dependencies.
Planned Impact
The investigators and partners have a long track record of delivering impact and pulling through ideas from research into exploitation.
COMPPACT will provide impact through:
- Shaping UK capability in video compression and delivery: It will deliver reference software that enables development
and evaluation. This will provide benefit to industry as well as to the research community, in providing an ability to evaluate and compare codec performance.
- Standardisation: We anticipate that the ideas and methods of COMPPACT will contribute to future video compression standardisation. This will provide new opportunities for video IC design companies in developing superior products. It will also provide benefits to the mobile communications industry through more efficient use of bandwidth, and to application developers and content providers in terms of being able to provide more immersive content at lower bit rates.
- Society: A major impact of this project will be on the wider public, in terms of providing high quality bandwidth-efficient video services, more engaging content representations and enabling new more immersive formats.
- The Bristol Creative Community: will benefit through local events and workshops hosted at the Watershed Arts Centre. We will also provide access by the local creative sector to capture, display and quality assessment facilities in the BVI studio, in particular to SMEs through the Bristol Media Network and the BBC-Anchor Consortium.
COMPPACT will provide impact through:
- Shaping UK capability in video compression and delivery: It will deliver reference software that enables development
and evaluation. This will provide benefit to industry as well as to the research community, in providing an ability to evaluate and compare codec performance.
- Standardisation: We anticipate that the ideas and methods of COMPPACT will contribute to future video compression standardisation. This will provide new opportunities for video IC design companies in developing superior products. It will also provide benefits to the mobile communications industry through more efficient use of bandwidth, and to application developers and content providers in terms of being able to provide more immersive content at lower bit rates.
- Society: A major impact of this project will be on the wider public, in terms of providing high quality bandwidth-efficient video services, more engaging content representations and enabling new more immersive formats.
- The Bristol Creative Community: will benefit through local events and workshops hosted at the Watershed Arts Centre. We will also provide access by the local creative sector to capture, display and quality assessment facilities in the BVI studio, in particular to SMEs through the Bristol Media Network and the BBC-Anchor Consortium.
Publications
Zhang F
(2019)
Rate-Distortion Optimization Using Adaptive Lagrange Multipliers
in IEEE Transactions on Circuits and Systems for Video Technology
Zhang F
(2013)
Quality assessment methods for perceptual video compression
Mercer Moss F
(2016)
On the Optimal Presentation Duration for Subjective Video Quality Assessment
in IEEE Transactions on Circuits and Systems for Video Technology
Zhang Y
(2016)
High Dynamic Range Video Compression Exploiting Luminance Masking
in IEEE Transactions on Circuits and Systems for Video Technology
Zhang Y
(2016)
High Dynamic Range Video
Zhang F
(2016)
HEVC enhancement using content-based local QP selection
Vigars R
(2013)
Context-based video coding
Zhang F
(2018)
BVI-HD: A Video Quality Database for HEVC Compressed and Texture Synthesized Content
in IEEE Transactions on Multimedia
Description | This project has developed a novel means of video compression based on texture warping and synthesis. Instead of encoding whole images or prediction residuals after translational motion estimation, our algorithm employs a perspective motion model to warp static textures and utilises texture synthesis to create dynamic textures. Texture regions are segmented using features derived from the complex wavelet transform and further classified according to their spatial and temporal characteristics. The project has also developed low complexity perceptual quality metrics that have the potential to be used in the coding loop to prevent warping and synthesis artefacts, as well as outside of it. The new perceptual video quality metric (PVM) out performs all previously published work. This work is now been published in IEEE Trans CSVT. The proposed algorithm has been integrated into an H.264 video coding framework. The results show significant bitrate savings, of up to 60% compared with H.264 at the same objective quality (based on AVM) and subjective scores. It is currently being integrated into the new HEVC video coding standard. COMPPACT is currently undertaking further data collection exercises including various textural content acquired at high spatial resolution and high frame rates. Results of enhanced Rate Quality Optimisation have also been researched that have demonstrated significant improvements over the latest coding standard - HEVC. Content-adaptive Langragian multiplier selection methods have also been generated that improve upon HEVC performance by 3%. This work led to the later work in the EPSRC Platform Grant - Vision for the Future - on Spatio temporal resampling combined with superresolution upsampling as a basis for perceptual video compression. This has been patented and submitted to MPEG. |
Exploitation Route | A number of important collaborations have already arisen from this work where the findings and datasets are already being used. These include: i) an EU Marie Curie Training Network, collaboratively with Fraunhofer HHI Berlin, Univ. Aachen, Univ. Nantes, BBC, Microsoft, Technicolor, Netflix, Google and Purdue Univ. This project builds on COMPPACT in the area of perceptual video coding and quality metrics and is targeting the next generation of video coding standards. ii) An EPSRC Platform grant awarded to Bristol Vision Institute (Bull PI), 'Vision for the Future'. A key theme in this is visual immersion where compression techniques will be investigated that achieve compression ratios exceed 1000:1 while preserving the immersive properties of the format. Collaborators include the Academy of Motion Picture Arts and Sciences, Aardman Animations, BBC, and ARRI. iii) A joint facility, 'The Bristol BBC Immersive Technology Laboratory has been established that is investigating compression for high frame rate and High Dynamic Range content. iv) ViSTRA codec has been submitted for consideration by MPEG and is the basis for collaboration with Netflix. |
Sectors | Aerospace Defence and Marine Creative Economy Digital/Communication/Information Technologies (including Software) Education Electronics Healthcare Leisure Activities including Sports Recreation and Tourism Manufacturing including Industrial Biotechology Culture Heritage Museums and Collections Transport |
URL | http://www.bristol.ac.uk/vi-lab/projects/parametricvideocompression/ |
Description | The work of this project is likely to only have real impact during the next stage of video standardisation. There is currently a call for evidence being planned in MPEG, leading to a new H.266 standard in 2020. We have contributed to this. The ViSTRA codec, which has been developed based on COMPPACT and the EPSRC Platform Grant, Vision for the Future, has recently been submitted for consideration by MPEG in its latest call for Compression Beyond HEVC. Out of all submissions, it was ranked 10th and demonstrated one of the highest performance gains for a single tool. |
First Year Of Impact | 2017 |
Sector | Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Manufacturing, including Industrial Biotechology,Security and Diplomacy,Transport |
Impact Types | Societal Economic Policy & public services |
Description | EPSRC IAA ViSTRA |
Amount | £20,000 (GBP) |
Organisation | University of Bristol |
Sector | Academic/University |
Country | United Kingdom |
Start | 12/2017 |
End | 10/2018 |
Description | EPSRC Platform Grant |
Amount | £1,362,874 (GBP) |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 02/2015 |
End | 02/2020 |
Description | EU Marie Curie ITN |
Amount | £607,602 (GBP) |
Organisation | Marie Sklodowska-Curie Actions |
Department | Initial Training Networks (ITN) |
Sector | Charity/Non Profit |
Country | Global |
Start | 08/2013 |
End | 09/2017 |
Description | Netflix |
Amount | £50,000 (GBP) |
Organisation | Netflix, Inc. |
Sector | Private |
Country | United States |
Start | 03/2018 |
End | 03/2019 |
Description | YouTube Faculty research Award |
Amount | £40,000 (GBP) |
Organisation | YouTube |
Sector | Private |
Country | United States |
Start | 06/2017 |
End | 09/2020 |
Title | BV High frame rate database |
Description | Collection of high frame rate clips with associated metadata for testing and developing future immersive video formats |
Type Of Material | Database/Collection of data |
Year Produced | 2015 |
Provided To Others? | Yes |
Impact | None at present |
URL | http://data.bris.ac.uk/data/dataset/k8bfn0qsj9fs1rwnc2x75z6t7 |
Title | BVI Texture database |
Description | Collection of static and dynamic video textures for compression testing |
Type Of Material | Database/Collection of data |
Year Produced | 2015 |
Provided To Others? | Yes |
Impact | Used by several groups around the world |
URL | http://data.bris.ac.uk/datasets/1if54ya4xpph81fbo1gkpk5kk4/ |
Title | BVI-HFR |
Description | Bristol vision institute high frame rate video database, which contains 22 unique video sequences with frame rates up to 120Hz. |
Type Of Material | Database/Collection of data |
Provided To Others? | No |
Impact | First publicly available high frame rate video database |
URL | http://data.bris.ac.uk/data/dataset/k8bfn0qsj9fs1rwnc2x75z6t7 |
Description | BBC Immersive Technology Laboratory |
Organisation | British Broadcasting Corporation (BBC) |
Department | BBC Research & Development |
Country | United Kingdom |
Sector | Public |
PI Contribution | High Dynamic range coding optimisation for HEVC Perceptual video compression results REDUX database analytics |
Collaborator Contribution | Provision of REDUX Support for PhD students Collaboration on perceptual quantisation Secondment of BBC employees |
Impact | New method of perceptual quantisation for HDR HEVC Analysis of BBC archive in terms of feature classification |
Start Year | 2012 |
Description | Netflix collaboration |
Organisation | Netflix, Inc. |
Country | United States |
Sector | Private |
PI Contribution | Video codec research, perceptual metrics and dynamic optimisation. From 2022 research into intelligent tools for the AoM AV2 standard. |
Collaborator Contribution | Data set access, shared resources and expertise. |
Impact | Characterisation and enhancement of perceptual VMAF metric; performance comparisons AV1 vs HEVC. New Feature based Dynamic Optimisation Method. |
Start Year | 2018 |
Title | Video Processing Method (ViSTRA) |
Description | Optimisation of video codec using in loop perceptual metrics and superresolution upscaling |
IP Reference | P123219GB |
Protection | Patent application published |
Year Protection Granted | |
Licensed | No |
Impact | Submitted to MPEG Beyond HEVC |
Description | Keynote: IET ISP |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Keynote Lecture IET ISP- Perceptual Video coding |
Year(s) Of Engagement Activity | 2015 |