Programmable embedded platforms for remote and compute intensive image processing applications

Lead Research Organisation: Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences


Image processing is playing an increasingly important role in our lives whether this is the numerous sources of social provision e.g. TV, or the increased reliance on security to protect our everyday lives through the proliferation of security cameras in airports and town centres. There are also healthcare applications with increased need for 3-dimensional (3D) images such as in viewing 3D computerised tomography scans to provide much more intelligent treatment. In automotive applications, cameras are used for quality assurance in manufacture and situational awareness in use. In security applications, organisations are keen to have more intelligent views of scenes to highlight security risks and dangers. This has increased the amount of visual information that we process and store, and has placed increasing importance on the users' ability to process data where it is received, thus pushing for more intelligent image processing.
Whilst a lot of innovative work has been done to derive the algorithms to provide this intelligence, there is a clear need for suitable, high performance, lower power hardware to provide the processing as in many cases, these systems may be remote e.g. security cameras with limited interconnection. We could wait for technology evolutions to provide the increased performance as before, but the warnings on process variability below 45-nm CMOS technology suggest that this might not be forthcoming and implies an increased focus on novel processor architectures is required. Whilst multi-core and application specific processors such as graphical processing units (GPUs) have been proposed, the gains have been limited. In addition, the rapid developments in the acquisition and interpretation of images together with intelligent algorithmic development, have not been matched by sound software engineering principles to develop and transform code into hardware implementations efficient in speed, memory and power. In many cases, image sensors comprise simple processing engines which communicate to some central resource for further processing. For a lot of medical and security applications, there is a need for more intelligent image acquisition, multi-view video processing (merging many views into a more useful, higher-level representation) and more context-aware acquisition devices which are aware of the existence of other cameras which can contribute to the creation of the full scene. This requires a step change in how we design and program these systems.
Current FPGA technology such as the Xilinx Virtex-7 FPGA, offers a huge performance capability (over 6.7 Giga Multiply-Accumulate per second and up to 30 Terabits/s of memory bandwidth) and better power efficiency than GPUs. Currently FPGA solutions are created by aggregating powerful intellectual property (IP) cores together with soft cores, but the resulting performance is limited by the overall systems architecture and programmability is severely limited. Hence, there is a clear need to derive a FPGA system architecture that best matches the algorithmic requirements but that is programmable in software for a range of algorithms in the application domain. By considering the model of computation and programming model from the outset, we propose to create a highly powerful platform for a range of image processing algorithms. The proposal combines the FPGA processor design expertise in Queen's University (Woods), with the software language and compiler research (Michaelson) and image processing expertise (Wallace) at Heriot-Watt University. A key aspect is to ensure close interaction between the processor development and software languages and representation, in order to ensure the creation of a processor architecture configuration that is programmable in software. The research looks to radically alter the design of front end image processing systems by offering the performance of FPGA solutions with the programmability of processor solution
Description Reconfigurable hardware (Field Programmable Gate Arrays aka FPGAs) offer strong opportunities for efficient implementation of complex algorithms, in particular for image processing, but are very hard for non-experts to deploy effectively. We have elaborated an approach based in a very high level domain specific language (DSL) where the programmer may focus on expressing the algorithm and a rich toolset will then aid with mapping it to the FPGA. Our key outcomes are: a) a methodology for representation of Streaming Image Processing algorithms by dataflow abstractions, allowing code transformation and restructuring leading to improved performance on both FPGA and CPU architectures b) the RIPL DSL c) substantial exemplars across the spectrum of image processing in RIPL d) a machine assisted tool for exploring different configurations of RIPL programs driven by precise performance information e) design and implementation of a smart camera architecture f) evaluation of our approach on real-time image processing algorithms on an FPGA against traditional hand-crafted code g) dissemination to a wide academic and industrial audience h) foundation of a growing international workshop series on DSLs, now in its 4th year.
Exploitation Route Our framework for resource driven refactoring based on dataflow should be more widely applicable to other architectures (e.g. many-core CPU, GPU) and to heterogeneous platforms that combine these.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Electronics,Energy,Transport

Description As part of our ongoing programme investigating the implementation of computer imaging and vision algorithms in embedded hardware, we have presented our findings in a number of collaborative industry-academia themed and knowledge transfer meetings under the auspices of the "Signal Processing for the Information Age" programme. Industrial partners include Mathworks, Seebyte, Atlas Elektronik, Cubicon, ADS, Kaon, BAE Systems, Leonardo, Thales, Qinetiq, and Roke Manor.
First Year Of Impact 2018
Sector Aerospace, Defence and Marine
Description Higher Education Impact Fellowship (HEIF) on low power and accelerated image processing hardware development
Amount £4,500 (GBP)
Organisation Sheffield Hallam University 
Sector Academic/University
Country United Kingdom
Start 06/2017 
End 06/2018
Description Match funded PhD studentship on 'Domain specific optimisations for real-time image processing on heterogeneous hardware'
Amount £143,000 (GBP)
Organisation ST Microelectronics 
Sector Private
Country Switzerland
Start 09/2019 
End 09/2022