Operational Refinement of Computation for Multimedia Coding Systems

Lead Research Organisation: University College London
Department Name: Electronic and Electrical Engineering

Abstract

Multimedia coding systems today cannot provide seamless quality degradation under degraded system resources. For example, if one watches a video on a portable video player, or is in the middle of a very important phone call, and this is draining the system resources (battery), current systems do not allow for seamless trade-offs in visual (audio) quality vs battery life (computation). Today the user is practically facing the on/off situation of the digital world, while one would strongly opt for an analogue world, where energy or computational resources (complexity) are traded off with multimedia quality (e.g. visual or audible distortion).We propose to fundamentally alter the way conventional multimedia coding algorithms are computed based on a new paradigm that we call Operational Refinement of Computation for Multimedia Coding Systems . The key principle is based on altering the realization of multimedia coding algorithms to enable the new principle of incremental refinement of computation: under a refinement of the multimedia information (e.g. images/video/audio), the algorithm computation refines the previously-computed result thereby leading to incremental computation of the output. The incremental processing or reconstruction of the input/output multimedia signals enables three key advantages in comparison to existing systems. Firstly, complexity-distortion trade-offs can be formulated since every refinement layer improves upon the quality of the output result (reduces distortion) at the cost of additional complexity. Secondly, each refinement input/output layer typically consists of data with limited dynamic-range (e.g. single-bit precision data). Hence, the complexity of the processing tasks can be modelled more accurately in function of the source statistics. Thirdly, each refinement layer can be scheduled in a different part of the implementation architecture and the computation of all layers can be parallelized. This is expected to increase the execution speed and hardware utilization significantly.This proposal comes at an excellent time. There has been a flurry of research on novel sampling and capturing devices that merge successive-approximation based analogue-to-digital converters with image sensors at the pixel or sample level. This enables the sample-based, or bitplane-based capturing of the input multimedia data. At the same time, very recent results demonstrated that image displays enabling the incremental refinement of a large number of luminance shades without flicker are possible. This enables the incrementally-produced output to be directly consumed by the display monitor. These novel developments in circuit theory and design seem very promising in solving the capturing and display aspects for systems that process the input data incrementally.In summary, conventional systems provide an all or nothing multimedia representation; the computation cannot be interrupted arbitrarily when resources become unavailable and retrieve a meaningful approximation of the final result. Contrasting the existing paradigm, we propose to investigate, for the first time, a new category of best-effort signal processing and multimedia systems. Applications of this type of systems are in all environments where resources may bescarce or uncertain due to environmental constraints, based on user choice, or, finally, by construction. Examples are:* portable multimedia systems with limited energy resources,* resource-constrained adaptive surveillance or monitoring applications with always on features,* fault tolerant multimedia algorithm and system design, and* progressive pricing schemes and progressive upgrades for quality-upgradeable hardware.

Publications

10 25 50
publication icon
Anam M (2012) Throughput Scaling Of Convolution For Error-Tolerant Multimedia Applications in IEEE Transactions on Multimedia

publication icon
Anastasia D (2010) Software designs of image processing tasks with incremental refinement of computation. in IEEE transactions on image processing : a publication of the IEEE Signal Processing Society

publication icon
Anastasia D (2010) Linear Image Processing Operations With Operational Tight Packing in IEEE Signal Processing Letters

publication icon
Andreopoulos Y (2010) Prediction-Based Incremental Refinement for Binomially-Factorized Discrete Wavelet Transforms in IEEE Transactions on Signal Processing

publication icon
Andreopoulos Y (2008) Incremental Refinement of Computation for the Discrete Wavelet Transform in IEEE Transactions on Signal Processing

publication icon
Andreopoulos Y (2008) Incremental refinement of image salient-point detection. in IEEE transactions on image processing : a publication of the IEEE Signal Processing Society

publication icon
Verdicchio F (2011) Distortion estimates for adaptive lifting transforms with noise in Image and Vision Computing

 
Description Multimedia coding systems today cannot provide seamless quality degradation under degraded system resources. For example, if one watches a video on a portable video player, or is in the middle of a very important phone call, and this is draining the system resources (battery), current systems do not allow for seamless trade-offs in visual (audio) quality vs battery life (computation). Today the user is practically facing the on/off situation of the digital world, while one would strongly opt for an analogue world, where energy or computational resources (complexity) are traded off with multimedia quality (e.g. visual or audible distortion).



This project proposed ways to fundamentally alter the way conventional multimedia coding algorithms are computed based on a new paradigm called: "Operational Refinement of Computation for Multimedia Coding Systems".



The key principle is based on altering the realization of multimedia coding algorithms to enable the new principle of incremental refinement of computation: under a refinement of the multimedia information (e.g. images/video/audio), the algorithm computation refines the previously-computed result thereby leading to incremental computation of the output.



The incremental processing or reconstruction of the input/output multimedia signals enables three key advantages in comparison to existing systems. Firstly, complexity-distortion trade-offs can be formulated since every refinement layer improves upon the quality of the output result (reduces distortion) at the cost of additional complexity. Secondly, each refinement input/output layer typically consists of data with limited dynamic-range (e.g. single-bit precision data). Hence, the complexity of the processing tasks can be modelled more accurately in function of the source statistics. Thirdly, each refinement layer can be scheduled in a different part of the implementation architecture and the computation of all layers can be parallelized. This provides a significant increase in the execution speed and hardware utilization.
Exploitation Route The R&D efforts of ST Microelectronics(*) (associated with this research by an accompanying industrial grant) may lead to related products being launched by the company in the near future in the area of power-efficient, high-performance, video coding systems.



(*) STM also has an associated lab in Bristol, UK. This research was partially supported by an industrial grant from ST Microelectronics srl, (STM), Advanced Systems Technology (AST) Lab in Agrate, Italy(*). In return for the research support provided by STM, software demonstrating the distortion modelling approach derived by UCL was delivered to STM and was used for internal R&D efforts on high-efficiency Scalable Video Coding Systems. The AST Lab in Agrate, Italy, is very actively involved in developing such prototype systems that later on move to products sold by STM.



Other potential usages for the derived outcomes can be formed by the exploitation of the open-source demonstration projects available at the project webpage http://www.ee.ucl.ac.uk/~iandreop/ORIP.html

either for research purposes, or for industrial R&D in advanced multimedia systems.



Until the time of this writing, these projects have been downloaded by more than 50 different companies and Universities around the world, including the UK, Europe, Asia and the US.



(*) STM also has an associated lab in Bristol, UK
Sectors Digital/Communication/Information Technologies (including Software),Electronics

URL http://www.ee.ucl.ac.uk/~iandreop/ORIP.html
 
Description The original partnership with STMicroelectronics went ahead as planned in the letter of support provided at the submission stage (dated March 23, 2007), with the change of the Hosting Institution from Queen Mary University to UCL, due to the move of Dr. Andreopoulos to UCL. In return for the research support provided by STM, software demonstrating the distortion modelling approach derived by UCL was delivered to STM and was used for internal R&D efforts on high-efficiency Scalable Video Coding Systems. The AST Lab in Agrate, Italy, is very actively involved in developing such prototype systems that later on move to products sold by STM. The Research Associate who worked on this project (Dr. Fabio Verdicchio) was hosted in the AST lab for approximately 12 weeks, during the early stages of the project. Researchers from ST Microelectronics also visited UCL several times during the execution of the research to discuss progress. Finally, UCL researchers visited STM on two other occasions (2009 and 2010), to participate and present research work in the very successful "ST Streaming Day" event organized annually by ST Microelectronics.", e.g. see: http://stday2010.uniud.it/stday2010/papers.html In terms of published research outputs, this collaboration resulted in the key journal paper: F. Verdicchio and Y. Andreopoulos, "Distortion Estimates for Adaptive Lifting Transforms with Noise," Image and Vision Computing, to appear, at which the support of this grant and the grant from ST Microelectronics is acknowledged. The PI (Dr Andreopoulos) and STM continued to collaborate on related research work in the context of a follow-up project proposal for the FP7 framework. Overall, this collaboration was deemed very satisfactory from both sides, both because of the bilateral knowledge transfer and from the potential for exchanges of technologies and research outcomes in the future. Beyond impact in industry, related open-source software was released in the public domain via the project's website: http://www.ee.ucl.ac.uk/~iandreop/ORIP.html and two related testbeds were also released on Sourceforge: https://sourceforge.net/projects/unvedu/ https://sourceforge.net/projects/unv/ At the time of this writing, both of these outputs have been downloaded more than 4000 times from more than 50 countries across the globe (e.g., see download statistics on Sourceforge).
First Year Of Impact 2010
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Societal,Economic

 
Description PhD scholarship from the Commonwealth Scholarship Commission
Amount £79,584 (GBP)
Funding ID BDCA-2010-5 
Organisation Government of the UK 
Department Commonwealth Scholarship Commission
Sector Public
Country United Kingdom
Start 09/2010 
End 03/2013