Operational Refinement of Computation for Multimedia Coding Systems

Multimedia coding systems today cannot provide seamless quality degradation under degraded system resources. For example, if one watches a video on a portable video player, or is in the middle of a very important phone call, and this is draining the system resources (battery), current systems do not allow for seamless trade-offs in visual (audio) quality vs battery life (computation). Today the user is practically facing the on/off situation of the digital world, while one would strongly opt for an analogue world, where energy or computational resources (complexity) are traded off with multimedia quality (e.g. visual or audible distortion).We propose to fundamentally alter the way conventional multimedia coding algorithms are computed based on a new paradigm that we call Operational Refinement of Computation for Multimedia Coding Systems . The key principle is based on altering the realization of multimedia coding algorithms to enable the new principle of incremental refinement of computation: under a refinement of the multimedia information (e.g. images/video/audio), the algorithm computation refines the previously-computed result thereby leading to incremental computation of the output. The incremental processing or reconstruction of the input/output multimedia signals enables three key advantages in comparison to existing systems. Firstly, complexity-distortion trade-offs can be formulated since every refinement layer improves upon the quality of the output result (reduces distortion) at the cost of additional complexity. Secondly, each refinement input/output layer typically consists of data with limited dynamic-range (e.g. single-bit precision data). Hence, the complexity of the processing tasks can be modelled more accurately in function of the source statistics. Thirdly, each refinement layer can be scheduled in a different part of the implementation architecture and the computation of all layers can be parallelized. This is expected to increase the execution speed and hardware utilization significantly.This proposal comes at an excellent time. There has been a flurry of research on novel sampling and capturing devices that merge successive-approximation based analogue-to-digital converters with image sensors at the pixel or sample level. This enables the sample-based, or bitplane-based capturing of the input multimedia data. At the same time, very recent results demonstrated that image displays enabling the incremental refinement of a large number of luminance shades without flicker are possible. This enables the incrementally-produced output to be directly consumed by the display monitor. These novel developments in circuit theory and design seem very promising in solving the capturing and display aspects for systems that process the input data incrementally.In summary, conventional systems provide an all or nothing multimedia representation; the computation cannot be interrupted arbitrarily when resources become unavailable and retrieve a meaningful approximation of the final result. Contrasting the existing paradigm, we propose to investigate, for the first time, a new category of best-effort signal processing and multimedia systems. Applications of this type of systems are in all environments where resources may bescarce or uncertain due to environmental constraints, based on user choice, or, finally, by construction. Examples are:* portable multimedia systems with limited energy resources,* resource-constrained adaptive surveillance or monitoring applications with always on features,* fault tolerant multimedia algorithm and system design, and* progressive pricing schemes and progressive upgrades for quality-upgradeable hardware.