Distributed Heterogeneous Vertically Integrated Energy Efficient Data Centres

Lead Research Organisation: Queen's University of Belfast
Department Name: Electronics Electrical Eng and Comp Sci


Our world is in the midst of a "big data" revolution, driven by the ubiquitous ability to gather, analyse, and query datasets of unprecedented variety and size. The sheer storage volume and processing capacity required to manage these datasets has resulted in a transition away from desktop processing and toward warehouse-scale computing inside data centres. State-of-the-art data centres, employed by the likes of Google and Facebook, draw 20-30 MW of power, equivalent to 20,000 homes, with these companies needing many data centres each. The global data centre energy footprint is estimated at around 2% of the world's energy consumption and doubles every five years [33, 34]. Contemporary data centres have an average overhead of 90% [32], meaning that they consume up to 1.9 MW to deliver 1 MW of IT support; this is not cost-effective or environmentally sound. If the exponential data growth and processing capacity are to scale in the way that both the public and industry have come to rely upon, we must tackle the data centre energy crisis or face the reality of stagnated progress. With the semiconductor industry's inability to further lower operating voltages in processor and memory chips, the challenge is in developing technologies for large-scale data-centric computation with energy as a first-order design constraint.
The DIVIDEND project attacks the data centre energy efficiency bottleneck through vertical integration, specialisation, and cross-layer optimisation. Our vision is to present heterogeneous data centres, combining CPUs, GPUs, and task-specific accelerators, as a unified entity to the application developer and let the runtime optimise the utilisation of the system resources during task execution. DIVIDEND embraces heterogeneity to dramatically lower the energy per task through extensive hardware specialisation while maintaining the ease of programmability of a homogeneous architecture. To lower communication latency and energy, DIVIDEND leverages SoC integration and prefers a lean point-to-point messaging fabric over complex connection-oriented network protocols. DIVIDEND addresses the programmability challenge by adapting and extending the industry-led heterogeneous systems architecture programming language and runtime initiative to account for energy awareness and data movement. DIVIDEND provides for a cross-layer energy optimisation framework via a set of APIs for energy accounting and feedback between hardware, compilation, runtime, and application layers. The DIVIDEND project will usher in a new class of vertically integrated data centres and will take a first stab at resolving the energy crisis by improving the power usage effectiveness of data centres by at least 50%.
Description In the DIVIDEND project, we explored new approaches to model and apportion power consumption
between co-executing VMs (Virtual Machines). Our approach collects in runtime utilisation parameters
for each VM, such as CPU time, memory and IO usage, and estimates contribution of a VM to the total
system power consumption on the basis of these parameters. The experimental study shows that our
model precisely predicts VM power consumption with the mean absolute percentage error 3.5%. This
approach allows data centre service operators to account consumed energy for each customer in a
straightforward way, while the available alternative models are cpu-time utilisation based which is not
enough to infer about contribution of each VM to the total power consumption. In turn, we expect that
applying of the new energy accounting/pricing model will inadvertently make data centre users to
become aware of and improve power efficiency of running applications. As follow, in a long-term
perspective, we assume that the new energy accounting policy will help operators to reduce peak power
consumption of data centre.
The second part of this project involves energy and power profiling of software running on a
heterogeneous architecture. The AMD A-series processor (AMD A10-7870K) has been chosen as the base
platform for experimental study in the DIVIDEND project. This processor belongs to the generation of
accelerated processing units (APUs) which implements the HSA (Heterogeneous System Architecture ) by
combining multiple-core CPUs and discrete GPUs. Profiling of an application running on the
heterogeneous architecture gives insights into energy distribution between CPU and GPU parts of the
processor. Particularly, it enables understanding of how much energy is spent on execution of a program
kernel and how much energy is required to start this kernel and transfer data to GPU. The results of energy
profiling for an application running on the heterogeneous platform can help software engineers to
improve energy efficiency of this application by balancing load between CPU and GPU.
Integration of the developed profiling technologies with hardware and software solutions, including
Distributed HSA, Scale-Out NUMA, HSA for FPGA, provided by other partners in the DIVIDEND project will
result in an experimental heterogeneous distributed substrate and background knowledge highly
demanded by companies to build commodity energy efficient heterogeneous data centre.
Exploitation Route The findings of the DIVIDEND project have been used to explore the energy-efficiency of datacenter architectures that integrate multiple forms of computational accelerators (FPGAs, GPUs, and Dataflow Engines) within the context of the VINEYARD H2020 project and in collaboration with Maxeler. The findings of DIVIDEND are also underpinning two ongoing PhD projects at Queen's University Belfast. The first project explores software methods for deploying applications in Edge Computing environments where computational accelerators are enabled in both edge devices (such as mobile user devices, routers and switches) and Cloud servers. The second project explores virtual machine migration between Cloud providers based on cost, performance and energy-efficiency criteria.
Sectors Digital/Communication/Information Technologies (including Software),Electronics,Energy

Description The findings of this project have led to the proposal and award of a Knowledge Transfer Partnership project with Vox Financial Services. The partnership explores the use of blockchain technology for managing regulatory change.
First Year Of Impact 2018
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Economic

Description EPSRC ICT Delivery Planning Workshops
Geographic Reach National 
Policy Influence Type Participation in a national consultation
Description EU Horizon2020 Programme: Vineyard Project
Amount € 4,815,810 (EUR)
Funding ID 688540 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 02/2016 
End 01/2019
Description Horizon2020 Programme
Amount € 5,999,510 (EUR)
Funding ID H2020-732631 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 01/2017 
End 12/2020
Description Royal Society Wolfson Research Merit Award: Principles and Practice of Near-Data Computing
Amount £50,000 (GBP)
Funding ID WM150009 
Organisation The Royal Society 
Sector Academic/University
Country United Kingdom
Start 09/2015 
End 08/2020
Description Collaboration with ARM and Applied Micro on micro-servers operating with extended voltage/frequency margins 
Organisation ARM Holdings
Country United Kingdom 
Sector Private 
PI Contribution New resource management methods at the operating system & hypervisor levels for leveraging extended operating margins of 64-bit ARM processors to improve system-wide energy-efficiency in micro-servers.
Collaborator Contribution Donation of experimental boards from Applied Micro (XGene) and ARM (Juno).
Impact Ongoing development of variation-aware hypervisors with enhanced resilience, power management and performance management capabilities.
Start Year 2016
Description Collaboration with ARM on fine-grain energy accounting tools 
Organisation ARM Holdings
Country United Kingdom 
Sector Private 
PI Contribution Partnership with ARM on developing energy-efficient system software and methods for accurate and fine-grain energy accounting in the Linux operating system. Our research team has contributed new probabilistic models for energy accounting at time scales which are finer than the power sensing or sampling periods of on-chip or off-chip power sensing instruments.
Collaborator Contribution ARM has scoped this collaboration and contributed in an advisory role via a series of online and face-to-face meetings with the ALEA, ENPOWER and GEMSCLAIM project consortia.
Impact Thread-level energy accounting tools for ARM Big.Little platforms (e.g. Exynos) and their use in energy-aware scheduling and resource allocation on mobile devices are currently in implementation and preliminary evaluation stages.
Start Year 2016
Description Collaboration with IBM on disaggregated memory technologies and near-data computing 
Organisation IBM
Country United States 
Sector Private 
PI Contribution Our research team has contributed methods to manage data caching and placement on disaggregated memory architectures with near-data processing elements.
Collaborator Contribution IBM has contributed novel remote memory server infrastructures and near-data acceleration technologies.
Impact Materialised through an industrial placement of QUB research staff, this partnership is exploring designs to substantially improve the energy-efficiency of large memory systems, via the use of disaggregation of memory, RDMA-based networking to remote memory devices and near-data accelerators for in-situ, in-memory analytics.
Start Year 2015
Description Collaboration with Maxeler on integrating dataflow accelerators in Big Data software stacks 
Organisation Maxeler Technologies Inc
Department Maxeler Technologies
Country United Kingdom 
Sector Private 
PI Contribution Integration of Maxeler's dataflow engines into the Spark, Storm and other Big Data software stacks, in collaboration with Maxeler Technologies and STFC Hartree.
Collaborator Contribution Programming APIs for Maxeler dataflow accelerators.
Impact No outputs yet, extensions of Spark and Storm with streaming APIs using Maxeler dataflow engines are currently under design.
Start Year 2016
Title ALEA Energy Accounting Tool 
Description The ALEA profiler is a cross-platform statistical profiling tool for Linux, which provides time and energy profiling at the basic block level on Intel and ARM architectures (32 and 64 bit). Energy profiling is available for platforms with energy or power meters. Currently, ALEA supports all Intel platforms with enabled RAPL interface and ARM-based Odoroid-XU/Odroid-XU3 platforms. The tool can be used for profiling both sequential and multi-threaded applications. Energy and execution time accounting to source code is also supported for applications compiled with debugging information (DWARF). 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact ALEA is actively used by a number of groups in the UK and the US (e.g. Edinburgh, Lancaster, Virginia Tech, Old Dominion University) to perform targeted, energy-aware code optimisation on high-end computing systems. 
URL https://hpdc-gitlab.eeecs.qub.ac.uk/lmukhanov/alea-release.git