libhpc: Intelligent Component-based Development of HPC Applications

Lead Research Organisation: Imperial College London
Department Name: Dept of Computing


Libhpc aims build and demonstrate an environment for the component-based construction of HPC applications. This environment will enable method implementers to publish and share their contributions as multi-implementation adaptable components, shareable across domains and architectures, making them inherently more re-usable. End users will then be able to compose applications without reference to underlying machine or processor architectures. The aim of libhpc is to initiate a software development pattern for HPC that has knowledge capture and re-use at its core.Libhpc will be based on the combination of two long lines of research: that of Professor Sherwin in the development and application of spectral/hp element codes and that of Professor Darlington in the development of innovative application development methodologies. The project will take as its starting point the Nektar++ methods library and its application to the modelling of blood flow through an aortic arch. This application and the library will be re-factored to identify the necessary re-usable components and co-ordination forms required. These components will be designed and implemented, targeted at a variety of architectures, including, distributed memory clusters, shared memory clusters and GPU processors. A software methods repository with supporting meta-data will be constructed along with application-construction and mapping tools. This environment will then be used to re-express the aortic modelling application and map it systematically to a number of different architectures and machine configurations. These applications will be benchmarked to assess their performance and the improvement in human productivity and reusability gained.

Planned Impact

If successfully carried out we firmly believe that this project could have considerable long-lasting impact addressing, as it does, fundamental issues concerning the way HPC applications are developed and deployed. As we have discussed earlier we consider that there is a great amount of knowledge concerning HPC methods and their implementations that is currently not captured in a manner that allows it to be effectively re-used or re-deployed. Our framework allows the expertise and knowledge of experts in HPC methods and their implementations to be captured explicitly in a manner that allows it to be systematically re-used. Our method is a classic example of the machine-tool paradigm - invest effort early in the development cycle to develop methods that are generic and re-usable and, thus, save much more effort downstream when these methods are used. The development framework would provide a much more efficient way of building and using HPC applications, supporting re-use and re-application of methods across a variety of applications and application types. The framework would also provide a pedagogical vehicle to help teach, in structured way, numerical methods, solvers, their relationships and implementation techniques, a key technology or discipline - numerical methods or computational engineering - that is currently lacking such a vehicle. The principal beneficiaries are: 1. Domain scientists who use computational tools. Today this includes almost every scientist. They will benefit by being able to design and use extremely flexible applications which can be run on diverse computing utilities with improved performance. 2. Numerical methods implementers. They will have access to a method for publishing and sharing their work, which is otherwise typically buried in machine-bound implementation code. 3. Utility computing operators. This includes both research grid operators and commercial operators such as Amazon Cloud. libhpc's flexible format will allow researchers to deploy to on-demand grid configurations, thus increasing uptake of grid resources and increasing ROI, especially in institutional settings. 4. Machine vendors. By exposing metadata about their products, vendors will encourage purchase and use. 5. Scientific research as whole. This will benefit by having access to improved methods. With a more systematic building block approach, it is likely that methods development will accelerate.
Description Developing, using and maintaining software is still notoriously difficult, especially for complex HPC numerical modeling applications. The libhpc project tackled these problems based on an approach that combined several innovations: functional abstractions for both control (co-ordination forms) and data processing (functional components), alternative realisations or implementations for these abstractions and meta-data to record each component's capabilities and requirements. Method developers and implementers produce new components, annotate them with appropriate meta-data and place them in a software repository (which we now call a software Cloud). End-users build applications by composing abstract components. At application deployment/run-time an intelligent mapper examines the alternatives available, both component implementations and machine resources and produces an execution plan that optimises user-given criteria in terms of performance, cost or energy usage. The execution plant is then loaded and executed.
Exploitation Route The technology is generic, applicable to all types of applications and end-users, thus it is eminently usable in non-academic contexts. The developed libhpc methodology enables the development of software repositories (Software Clouds) comprising components, implementations and meta-data. Method developers and implementers could contribute new methods and implementations with appropriate meta-data to the repository and end-users would use these components to build an execute applications aided by the intelligent mapper. Thus libhpc extends the well-accepted model of code libraries but with much greater abstraction, generality and adaptability for the components.

These repositories would be built as Internet-available shared community resources. These could be provided as shared open-source resources or curated via some facility operator such as sourceforge or the Research Councils for their academic users. As such they would coordinate activities between method developers, end-users and facility operators, providing a modern, flexible HPC e-Infrastructure.

The libhpc project has successfully developed and applied such an infrastructure. A prototype framework has been built in both Python and Groovy that enables software repositories to be constructed and applications built and intelligently mapped as described above. A distributed environment, for both implementations hosting and execution, has been developed. This infrastructure has been demonstrated across a range of applications, particularly in bioinformatics and numerical modelling. A technology has been identified that would allow menu-driven user environments to be constructed for domain-specific application domains where the full scripting technology is not required but intelligent resource mapping is still desirable.

Application of these ideas to the Nektar++ spectral finite element code has made progress and a menu-driven framework (NekCloud) has been developed that allows Nektar++ to be used easily on Cloud IaaS environments.
Sectors Chemicals,Creative Economy,Digital/Communication/Information Technologies (including Software),Energy,Environment,Healthcare,Pharmaceuticals and Medical Biotechnology,Transport

Description Software, in one form or another, now underlies much activity in science, industry, commerce and public affairs. The difficulties in producing, using and maintaining software are well rehearsed. The technology built and demonstrated in libhpc has the potential to provide fundamental advances in the way complex software systems can be constructed, used and maintained, thus it has the potential to have a very considerable impact in the productivity, efficiency and reliability of software development and use in all these areas. Advances in the productivity and reliability with which complex software systems can be constructed and maintained. Beneficiaries: Society Contribution Method: Advances in fundamental technologies.
Sector Chemicals,Construction,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Energy,Environment,Healthcare,Pharmaceuticals and Medical Biotechnology,Transport
Impact Types Cultural,Economic

Description libhpc Stage II: A Long-term Solution for the Usability, Maintainability and Sustainability of HPC Software
Amount £726,567 (GBP)
Funding ID EP/K038788/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 01/2013 
End 09/2015
Title Libhpc Deployer Library 
Description The libhpc deployer is a Python software library for running batch-style HPC applications on different underlying computing platforms. At present the library supports running jobs on PBS-based HPC clusters, OpenStack private clouds and the Amazon EC2 public cloud service. The library uses YAML metadata for describing target platforms, applications and jobs. The library has a plugin-style design that allows support for additional target platforms to be added. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact The declarative style of the YAML metadata used to configure the library provides a simplified means for developers and end-user scientists and researchers to specify their jobs and switch much more easily between cluster and cloud platforms. For example, where users wish to undertake some jobs on a local cluster platform and take advantage of cloud infrastructure for other aspects of their workload, the libhpc deployer provides support for straightforwardly selecting a different target platform on a per-job basis. 
Title libhpc-cf: Libhpc Coordination Forms Library 
Description The libhpc-cf coordination forms library is a Python software library that allows creation of software component metadata and offers a set of "coordination forms" - functional style operators for coordinating the flow of control and data between the software components represented by component metadata. In practice, this means that developers can take existing software tools or libraries and wrap them as software components using the features provided by the libhpc-cf library. These components can then be linked to form more advanced processes by applying coordination forms to them. Coordination forms allow the specification of richer orchestration than traditional workflow languages and the library is extensible to allow the addition of further coordination forms. Coordination forms can vary from basic structures, such as specifying that a set of dependent components are processed sequentially or that a set of independent components may be processed in parallel, to advanced domain specific languages. A further key benefit of both components and coordination forms specified within the libhpc-cf library is that both can have multiple implementations allowing different approaches to be used to undertake a given task depending on the type of target computing platform to be used or the problem being addressed. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact The libhpc-cf library provides a modern Python implementation of a general approach pioneered by Darlington, Guo et al. in their 1995 paper "Functional Skeletons for Parallel Coordination". The library offers a demonstration of the differences and power of coordination forms when compared to existing workflow languages or systems. The ability to handle multiple implementations of software components and the orchestration processes used to control them is of particular use in modern heterogeneous computing environments.