Future-proof massively-parallel execution of multi-block applications

For many years, increasing the clock frequency of microprocessors has led to steady improvements in performance of computer applications. This gave an almost free performance boost to the speed of applications without having to re-write software for each new generation of processors. However, increasing the performance of processors in this manner led to an unsustainable increase in energy consumption. Thus, to gain higher performance chip developers now rely on multiple cores operating in parallel. The latest CPUs have up to 10 cores, each with a vector unit producing up to 8 single precision floating point results per clock cycle, while the latest graphics processors (GPUs) have up to 2688 much simpler cores operating in groups of 32.

This move into manycore computing has led to considerable hardware innovation, and it is likely that the next 10 years will see further rapid evolution in computer architectures. This poses huge challenges to application developers who naturally wish to concentrate on their engineering and scientific applications and how best to model them, without having to worry about the details of modern computer architectures. To address this, there are a range of efforts within scientific computing to develop high-level software packages or frameworks so that the application developer can specify what they want to be computed at a high level, and then the package takes care of the implementation details.

Building on prior EPSRC-funded research to develop a framework called OP2 for unstructured grid applications, this proposal aims to develop a future-proof extension called OPS to handle the needs of multi-block structured grid applications. Developers' applications can be written in FORTRAN or C, using a carefully-designed application programming interface (API), and then OPS generates customised code for the implementation on different hardware target platforms.

As well as customising for the different hardware, two other optimisation approaches will be adopted. One is the use of ``tiling'' to overlap the execution of parallel loops which are usually executed sequentially. This improves both performance and energy efficiency by reusing data within the cache, cutting down on the number of times data is moved between the processor and the main memory. This is something which is becoming increasingly important on modern architectures because the energy cost and time taken for data movement is much greater than for floating point operations.

The other optimisation is the use of run-time optimisation for applications which execute for a long time. The backend implementations are parameterised, with parameters controlling aspects such as the number of threads in a thread block, or the size of a ``tile'' in the tiling optimisation. The optimal values for these parameters are not known a priori, and it could significantly affect the performance. By dynamically varying the values, and timing the consequential changes in performance, we can implement heuristics to iteratively improve the parameter values during the execution.

The new OPS framework will be assessed, both for performance and ease-of-use, by applying it to two important academic CFD codes, ROTOR developed at Bristol by Prof. Chris Allen, and SBLI developed by at Southampton by Prof. Neil Sandham. As well as being important codes in their own right, these are also representative of the needs of other codes within CCP12 (Computational Engineering), the UK Turbulence Consortium, and the UK Applied Aerodynamics Consortium.

The Dominic Tildesley report on ``A Strategic Vision for UK e-Infrastructure'' includes many examples of the importance of computational modelling in a wide range of industries as well as in government, and it emphasises the importance of software, including the quote:

``This continuing growth in price/performance and power-efficiency comes at a cost: the new systems will be based on complex multi-core and accelerator-based architectures which are much more challenging to program than are today's systems, requiring a revolution in software design and development.''

The importance and challenge of manycore computing is also reflected in its inclusion as one of EPSRC's five main priorities within the ICT area.

Historically, the UK has played a leading role in computational modelling, relative to its size, particularly in areas such as aeronautical CFD and weather prediction. It is very important that this position is maintained, and one of the key ways of achieving this is to ensure that the computational modellers can spend their time developing better models, not worrying about the details of novel computer architectures.

At the same time, we need to train a new generation of scientific computing experts who do understand thoroughly the details of novel computer architectures and how best to exploit them. If £100M is being spent annually in the UK on HPC hardware (a very conservative estimate), then the cost savings are very substantial if software improvements such as tiling and run-time optimisation can deliver a factor 2 increase in performance.

Our research will develop open-source software giving support specifically for multi-block structured-grid applications in engineering and sceince, but more generally it will contribute towards the domain of manycore parallel computing which is vital for the health for the country's capability in computational modelling which underlies so much of modern engineering and science.