COMPPACT: Compression of Video using Perceptually Optimised Parametric Coding Techniques

Lead Research Organisation: University of Bristol
Department Name: Electrical and Electronic Engineering

Abstract

It is currently a very exciting and challenging time for video compression. The predicted growth in demand for bandwidth, especially for mobile services is driven largely by video applications and is probably greater now than it has ever been. There are four reasons for this:

(i) Recently introduced formats such as 3D and multiview, coupled with increasing dynamic range, spatial resolution and framerate, all require increased bit-rate to deliver improved immersion;
(ii) Video-based web traffic continues to grow and dominate the internet;
(iii) User expectations coninue to drive flexibility and quality, with a move from linear to non-linear delivery;
(iv) Finally the emergence of new services, in particular mobile delivery through 4G/LTE to smart phones. While advances in network and physical layer technologies will no doubt contribute to the solution, the role of video compression is also of key importance.

This research project is underpinned by the assumption that, in most cases, the target of video compression is to provide good subjective quality rather than to minimise the error between the original and coded pictures. It is thus possible to conceive of a compression scheme where an analysis/synthesis framework replaces the conventional energy minimisation approach. Such a scheme could offer substantially lower bitrates through reduced residual and motion vector coding.

The approach proposed will model scene content using combinations of waveform coding and texture replacement, using computer graphic models to replace target textures at the decoder. These not only offer the potential for dramatic improvements in performance, but they also provide an inherent content-related parameterisation which will be of use in classification and detection tasks as well as facilitating integration with CGI.

This has the potential to create a new content-driven framework for video compression. In this context our aim is to shift the video coding paradigm from rate-distortion optimisation to rate-quality modelling, where region-based parameters are combined with perceptual quality metrics to inform and drive the coding and synthesis processes. However it is clear that a huge amount of research needs to be done in order to fully exploit the method's potential and to yield stable and efficient solutions. For example, mean square error is no longer a valid objective function or measure of quality, and new embedded perceptually driven quality metrics are essential. The choice of texture analysis and synthesis models are also important, as is the exploitation of long-term picture dependencies.

Planned Impact

The investigators and partners have a long track record of delivering impact and pulling through ideas from research into exploitation.

COMPPACT will provide impact through:

- Shaping UK capability in video compression and delivery: It will deliver reference software that enables development
and evaluation. This will provide benefit to industry as well as to the research community, in providing an ability to evaluate and compare codec performance.

- Standardisation: We anticipate that the ideas and methods of COMPPACT will contribute to future video compression standardisation. This will provide new opportunities for video IC design companies in developing superior products. It will also provide benefits to the mobile communications industry through more efficient use of bandwidth, and to application developers and content providers in terms of being able to provide more immersive content at lower bit rates.

- Society: A major impact of this project will be on the wider public, in terms of providing high quality bandwidth-efficient video services, more engaging content representations and enabling new more immersive formats.

- The Bristol Creative Community: will benefit through local events and workshops hosted at the Watershed Arts Centre. We will also provide access by the local creative sector to capture, display and quality assessment facilities in the BVI studio, in particular to SMEs through the Bristol Media Network and the BBC-Anchor Consortium.
 
Description This project has developed a novel means of video compression based on texture warping and synthesis. Instead of encoding whole images or prediction residuals after translational motion estimation, our algorithm employs a perspective motion model to warp static textures and utilises texture synthesis to create dynamic textures. Texture regions are segmented using features derived from the complex wavelet transform and further classified according to their spatial and temporal characteristics.

The project has also developed low complexity perceptual quality metrics that have the potential to be used in the coding loop to prevent warping and synthesis artefacts, as well as outside of it. The new perceptual video quality metric (PVM) out performs all previously published work. This work is now been published in IEEE Trans CSVT.

The proposed algorithm has been integrated into an H.264 video coding framework. The results show significant bitrate savings, of up to 60% compared with H.264 at the same objective quality (based on AVM) and subjective scores. It is currently being integrated into the new HEVC video coding standard.

COMPPACT is currently undertaking further data collection exercises including various textural content acquired at high spatial resolution and high frame rates.

Results of enhanced Rate Quality Optimisation have also been researched that have demonstrated significant improvements over the latest coding standard - HEVC. Content-adaptive Langragian multiplier selection methods have also been generated that improve upon HEVC performance by 3%. This work led to the later work in the EPSRC Platform Grant - Vision for the Future - on Spatio temporal resampling combined with superresolution upsampling as a basis for perceptual video compression. This has been patented and submitted to MPEG.
Exploitation Route A number of important collaborations have already arisen from this work where the findings and datasets are already being used. These include:
i) an EU Marie Curie Training Network, collaboratively with Fraunhofer HHI Berlin, Univ. Aachen, Univ. Nantes, BBC, Microsoft, Technicolor, Netflix, Google and Purdue Univ. This project builds on COMPPACT in the area of perceptual video coding and quality metrics and is targeting the next generation of video coding standards.
ii) An EPSRC Platform grant awarded to Bristol Vision Institute (Bull PI), 'Vision for the Future'. A key theme in this is visual immersion where compression techniques will be investigated that achieve compression ratios exceed 1000:1 while preserving the immersive properties of the format. Collaborators include the Academy of Motion Picture Arts and Sciences, Aardman Animations, BBC, and ARRI.
iii) A joint facility, 'The Bristol BBC Immersive Technology Laboratory has been established that is investigating compression for high frame rate and High Dynamic Range content. iv) ViSTRA codec has been submitted for consideration by MPEG and is the basis for collaboration with Netflix.
Sectors Aerospace

Defence and Marine

Creative Economy

Digital/Communication/Information Technologies (including Software)

Education

Electronics

Healthcare

Leisure Activities

including Sports

Recreation and Tourism

Manufacturing

including Industrial Biotechology

Culture

Heritage

Museums and Collections

Transport

URL http://www.bristol.ac.uk/vi-lab/projects/parametricvideocompression/
 
Description The work of this project is likely to only have real impact during the next stage of video standardisation. There is currently a call for evidence being planned in MPEG, leading to a new H.266 standard in 2020. We have contributed to this. The ViSTRA codec, which has been developed based on COMPPACT and the EPSRC Platform Grant, Vision for the Future, has recently been submitted for consideration by MPEG in its latest call for Compression Beyond HEVC. Out of all submissions, it was ranked 10th and demonstrated one of the highest performance gains for a single tool.
First Year Of Impact 2017
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Manufacturing, including Industrial Biotechology,Security and Diplomacy,Transport
Impact Types Societal

Economic

Policy & public services

 
Description EPSRC IAA ViSTRA
Amount £20,000 (GBP)
Organisation University of Bristol 
Sector Academic/University
Country United Kingdom
Start 12/2017 
End 10/2018
 
Description EPSRC Platform Grant
Amount £1,362,874 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 02/2015 
End 02/2020
 
Description EU Marie Curie ITN
Amount £607,602 (GBP)
Organisation Marie Sklodowska-Curie Actions 
Department Initial Training Networks (ITN)
Sector Charity/Non Profit
Country Global
Start 08/2013 
End 09/2017
 
Description Netflix
Amount £50,000 (GBP)
Organisation Netflix, Inc. 
Sector Private
Country United States
Start 03/2018 
End 03/2019
 
Description YouTube Faculty research Award
Amount £40,000 (GBP)
Organisation YouTube 
Sector Private
Country United States
Start 06/2017 
End 09/2020
 
Title BV High frame rate database 
Description Collection of high frame rate clips with associated metadata for testing and developing future immersive video formats 
Type Of Material Database/Collection of data 
Year Produced 2015 
Provided To Others? Yes  
Impact None at present 
URL http://data.bris.ac.uk/data/dataset/k8bfn0qsj9fs1rwnc2x75z6t7
 
Title BVI Texture database 
Description Collection of static and dynamic video textures for compression testing 
Type Of Material Database/Collection of data 
Year Produced 2015 
Provided To Others? Yes  
Impact Used by several groups around the world 
URL http://data.bris.ac.uk/datasets/1if54ya4xpph81fbo1gkpk5kk4/
 
Title BVI-HFR 
Description Bristol vision institute high frame rate video database, which contains 22 unique video sequences with frame rates up to 120Hz. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact First publicly available high frame rate video database 
URL http://data.bris.ac.uk/data/dataset/k8bfn0qsj9fs1rwnc2x75z6t7
 
Description BBC Immersive Technology Laboratory 
Organisation British Broadcasting Corporation (BBC)
Department BBC Research & Development
Country United Kingdom 
Sector Public 
PI Contribution High Dynamic range coding optimisation for HEVC Perceptual video compression results REDUX database analytics
Collaborator Contribution Provision of REDUX Support for PhD students Collaboration on perceptual quantisation Secondment of BBC employees
Impact New method of perceptual quantisation for HDR HEVC Analysis of BBC archive in terms of feature classification
Start Year 2012
 
Description Netflix collaboration 
Organisation Netflix, Inc.
Country United States 
Sector Private 
PI Contribution Video codec research, perceptual metrics and dynamic optimisation. From 2022 research into intelligent tools for the AoM AV2 standard.
Collaborator Contribution Data set access, shared resources and expertise.
Impact Characterisation and enhancement of perceptual VMAF metric; performance comparisons AV1 vs HEVC. New Feature based Dynamic Optimisation Method.
Start Year 2018
 
Title Video Processing Method (ViSTRA) 
Description Optimisation of video codec using in loop perceptual metrics and superresolution upscaling 
IP Reference P123219GB 
Protection Patent application published
Year Protection Granted
Licensed No
Impact Submitted to MPEG Beyond HEVC
 
Description Keynote: IET ISP 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Keynote Lecture IET ISP- Perceptual Video coding
Year(s) Of Engagement Activity 2015