ABC: Adaptive Brokerage for the Cloud

The answer to both these questions is usually 'No' even by those familiar with cloud computing. Such uncertainty is caused by 3 main reasons:

1) A bewildering choice of service offerings by the many service providers (e.g. Amazon, Google, Microsoft, etc.).
2) Difficulty in comparison between offered services due to non-uniform description of specifications.
3) High time and monetary costs associated with continuous monitoring of the services on offer from various providers.

This project will introduce more certainty into the selection of cloud services through the introduction of a smart and continuously adaptive cloud broker. The broker will act as an intermediary between end users and cloud service providers in order to enhance service delivery and service value. This brokerage service will be designed to manage heterogeneous cloud offerings, including public, private or hybrid environments. Such brokerage will open up an entirely new multi-cloud marketplace, allowing applications to be simply deployed to the optimal provider and resource type, reducing complexity, vendor lock-in and computational running costs.

Our research will result in a number of fundamental contributions to the cloud computing field. First we will address the problem of how to define, schedule and enforce user-defined Service Level Objectives (SLOs): high-level intentions, which specify the desired end goal of a deployment for applications that span multiple cloud providers with complex inter-dependencies. This will allow users to focus on what (e.g., failure tolerance) needs to be achieved, rather than low-level specifics about how (e.g., deploy to Amazon compute optimised VM) applications are deployed. This automation will in turn help abstract many of the complexities associated with low-level configuration from the user.

Second, we will develop novel lightweight container-based benchmarking techniques, which can gather cloud-level performance metrics in near real-time in a multi-cloud environment. These techniques will be general in scope and allow users to obtain a near real time perspective of the 'weather', or current state across a range of cloud providers.

Third, we will develop adaptive machine learning strategies for the autonomic and pro-active management of cloud-based applications. The application of machine learning will aid decisions about which providers and resource configuration meet the requirements specified in the SLO, how these trade off against cost, and when to redeploy to different providers, or instance types based on active management, etc.

We are confident that this project will address an urgent and fundamental question: how to leverage cloud infrastructure to quickly, cheaply and efficiently perform vital computational workloads. Solving this problem is crucial to the UK digital economy, which is increasingly reliant on the cloud. The developed smart brokerage framework will enable digital economy stakeholders to optimise their use of cloud resources. This is beneficial to all areas of business, including start-ups and micro-businesses who can benefit greatly from the flexibility created by platform independence and adaptive management strategies.

This proposal is a careful balance between cutting edge research with significant academic impact, together with a pathways to impact strategy targeting economic, societal and people impact. Our multi-faceted impact strategy contains the following key elements:

1) Collaborating with key companies from the digital economy sector both in terms of evaluating the cloud brokerage technology and also providing an important route to technology transfer (targeting economic impact). More specifically, we will collaborate with two contrasting companies, Adobe and Satalia who bring use cases offering complete coverage of evaluating the level of platform independence offered alongside the level of optimisation in terms of deployment, management and execution in a range of different multi-cloud environments. To complement this, we will also reach out to micro-businesses and SMEs through established networks in Scotland and the North-West of England (using SICSA and the Knowledge Business Centre at Lancaster University).

2) Seeking the formation of a startup company and investigating potentially novel business models as an intrinsic aspect of the science targeting economic impact). We see real potential in developing a commercialisation strategy in parallel with the research programme as there are currently no comprehensive cloud brokerage services that allow services to be traded as a commodity.

3) Working with partners from the science community again in terms of providing a comprehensive evaluation of the approach and providing an additional technology transfer pathway (targeting societal impact). The goals of this work are to i) evaluate our approach and its effectiveness in supporting the scientific community, and ii) subsequently transfer knowledge on the use of cloud computing in science and, more specifically, the use of cloud brokerage in providing platform independence and optimising the use of cloud resources according to experimental constraints. We will work closely with two contrasting partners, providing coverage of key issues in computational science, i.e. the Centre for Ecology and Hydrology and colleagues working on astronomy at St Andrews.

4) Operating a public engagement programme, employing a range of mechanisms including press releases, social media and podcasts to reach wide audiences (targeting societal impact). We will work through existing forums including the Edinburgh TechMeetUp, SICSA DemoFest and KBC Technology Showcases and also plan to use Compucast (a monthly computer science podcast) to reach a general technical and scientific audience.

5) Developing people with advanced skills in cloud computing/management, an area of acute shortage in terms of supporting the digital economy (targeting people impact). This proposal will help address this acute shortage by training post-docs with internationally leading skill-sets in this area, ready for positions of leadership and influence in academia or industry. This development of people in terms of skills and knowledge is an additional important impact of the project, and this will be enhanced through pro-actively seeking means of linking student project work to the programme of research.