Multi-layered abstractions for PDEs

Lead Research Organisation: Imperial College London
Department Name: Computing

Abstract

Partial differential equations (PDEs) are the mathematical basis ofvast areas of science. From fluid problems such as flow past anaircraft or the circulation of the worlds oceans to structuralproblems in bridges or inside the bones of a body and even in theworld of financial options pricing, PDEs are the key description. Itis therefore unsurprising that computing solutions to PDEs consumes amassive proportion of high performance computing resources in bothindustry and academia.Hitherto, the software which solves PDEs has been written for specificsorts of hardware by people who had to be experts both in the sciencethey were modelling and in programming. This is an expensive anderror-prone process and, because it occurs in an informal way inengineering and science departments, has suffered from a lack of inputfrom modern computer science. However, what makes this proposalcritical for computational science are the large changes which areoccuring in hardware technology. The emergence of new massivelyparallel platforms, such as graphical processing units (GPUs) andmulticore CPU systems is requiring very different approaches tolow-level programming and, importantly, different approaches fordifferent hardware platforms. It also requires new research into thebest optimisation strategies for these platforms. The current approachto writing scientific software will not deliver the changes that arerequired within the industrial and government funding resources available.This project will break this paradigm by using multiple layers ofabstraction to present to scientists and engineers a way ofprogramming PDE problems which is independent of the hardware and tocomputer scientists, a program generation space in which they cangenerate optimal algorithms for PDEs on different hardware platformswithout becoming embroiled in domain science. In particular, we willdevelop a hardware abstraction layer (OP2) which will allow programsto be written by explicitly stating what is parallel. The OP2 layersoftware will then translate this into low-level code which utilisesthe parallelism in a particular hardware platform. For developersworking with the finite element method (FEM), an importantcomputational technique for PDEs, we will present an additional layerwhich will enable them to write the finite element problem in a highlevel language, UFL. This will then be translated to OP2 and thencedown to the hardware layer.We will demonstrate the effectiveness of this approach with two PhDprojects. One will look into the dispersion of polutants in urbansettings while the other will study the stress properties of surgicalbone implants. These are both important scientific questions whichcurrently suffer from a lack of resolution and therefore need to beable to employ the latest high performance hardware.

Planned Impact

This is a high-impact proposal with a direct route to applications both in science and in industry. The potential for impact lies in supporting new applications and new levels of detail and accuracy in computational modelling. Although scientific and engineering computation is not a large part of the UK economy, is has disproportionate impact as an enabling capability across many sectors of industry and in academic research. We believe that, if successful, this project will transform the way PDE codes are written - certainly within the finite-element computational fluid dynamics community and potentially beyond. Our strategy to maximise this impact is to begin by working closely with a small number of close collaborators who are already showing a high level of commitment to this agenda. Once we have demonstrated the value of the technology there, we will invite these key stakeholders to present the results of their experience in community engagement events. Early adopters will include Fluidity (a major academic adaptive, unstructured-mesh finite-element CFD code being developed within Imperial's AMCG group, VOX-FE (a voxel-based finite-element code developed specifically for the structural analysis of bone by Hull's MBE group), and HYDRA (the primary CFD at Rolls Royce, and their largest consumer of CPU cycles). Uptake is recognised as a major management risk for the project. Our primary goal is to embed our technology in major industrial and science codes through existing relationships. We plan to work through the Centre for Fluid Mechanics Simulation (CFMS) and the FEniCS user group to reach further afield, by inviting the owners of these codes to report on their experience. Building on these relationships we will organise workshops to engage with the user community at the end of each year (coordinated with software releases), for which a budget allowance has been made. Further details are provided in Section 5 of the Case for Support and in our Impact Plan.
 
Description We have learned a great deal about how to build really powerful software tools to support scientists who use simulation to model physical systems.

The key idea is abstraction: the design of the representation of the problem so that analysis, transformation and synthesis of high-performance code is easy.

The original idea of the project was to explore this idea with respect to how parallel computation on the mesh is represented. This idea ("PyOP2") has proven exceptionally powerful: we have used it, for example, to demonstrate, for the first time, how to automatically combine different loops over the mesh even when they have very complex mesh-dependent dependencies.

We have discovered two other important levels at which new abstractions help simplify, and thus enable, complicated optimisations. One ("COFFEE") focuses on the local computational loop nest at each element of the mesh, and has enabled us to approach theoretically-optimal reductions in the amount of work needed, while also helping to exploit vector instructions in modern processor architectures.

The other new abstraction in the hierarchy is more recent, and builds on work funded by this grant: we have discovered powerful new ways to simplify the code generation process. This "two-stage form compiler" yields dramatically simpler implementation than previous work on automating the finite element method, and exposes new optimisations exploiting the structure of complicated forms found in sophisticated physical models.
Exploitation Route The fruits of this research have been embodied in a major open-source software project, Firedrake (http://firedrakeproject.org). Firedrake has an extremely active group of developer and users and is in continuous development. In particular, it is becoming central to continuing research projects in weather, climate, tidal renewables and other geophysical simulation projects in both science and engineering.

Meanwhile our research publications have carefully identified the properties and the strategies needed for designing abstractions, or intermediate representations as they are called in the compiler community, in ways which are independent of their embodiment in particular software, and thus transferable to other contexts.
Sectors Aerospace, Defence and Marine,Energy,Environment,Healthcare

URL http://firedrakeproject.org
 
Description Non-academic impacts are currently at a fairly early stage but are in active, continuing and funded development towards applications. (1) Firedrake is being used to develop new discretisations for improved modelling of the dynamics of the atmosphere, feeding into collaborative work with the UK Met Office on the next-generation weather and climate models (funded by the NERC/Met Office GungHo and GungHo phase II projects). (2) Firedrake is being used to develop new modelling technology for multi-scale interactions between sub-metre scale installations and ocean currents over tens of kilometres is crucial for assessing environmental impacts of coastal engineering projects and tidal energy systems, and for optimisation to minimise project costs or maximise profits. This work is supported by EPSRC (EP/M011054/1) in collaboration with major industry players.
First Year Of Impact 2015
Sector Aerospace, Defence and Marine,Environment
Impact Types Economic

 
Description A new simulation and optimisation platform for marine technology
Amount £434,711 (GBP)
Funding ID EP/M011054/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2015 
End 02/2018
 
Description Platform: Underpinning Technologies for Finite Element Simulation
Amount £1,287,360 (GBP)
Funding ID EP/L000407/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 11/2013 
End 10/2018
 
Title Firedrake 
Description Firedrake is an automated system for the portable solution of partial differential equations using the finite element method (FEM). Firedrake enables users to employ a wide range of discretisations to an infinite variety of PDEs and employ either conventional CPUs or GPUs to obtain the solution. 
Type Of Technology Software 
Year Produced 2013 
Open Source License? Yes  
Impact Firedrake has a growing community of users with whom we are working closely. Some of the resulting papers are listed at http://firedrakeproject.org/publications.html. 
URL http://www.firedrakeproject.org
 
Title OP2 
Description OP2 is framework for the execution of unstructured grid applications on clusters of GPUs or multi-core CPUs. Although OP2 is designed to look like a conventional library, the implementation uses source-source translation to generate the appropriate back-end code for the different target platforms. 
Type Of Technology Software 
Year Produced 2012 
Open Source License? Yes  
Impact OP2 is being adopted by Rolls Royce to support GPU and multicore acceleration of their flagship HYDRA fluid dynamics software. PyOP2, which is developed from OP2's concepts, is a key abstraction in the Firedrake Project (www.firedrakeproject.org). 
URL http://www.oerc.ox.ac.uk/projects/op2
 
Title PyOP2 
Description PyOP2 is a domain-specific language (DSL) for the parallel executions of computational kernels on unstructured meshes or graphs. PyOP2 is a key enabling abstraction on which the Firedrake Project is built. It is based on concepts from OP2 (https://github.com/OP2/OP2-Common, http://www.oerc.ox.ac.uk/projects/op2) but is implement in Python and entirely at runtime, rather than as a source-to-source compiler-time tool. It includes substantial extensions and optimisations, mainly to support finite element applications. 
Type Of Technology Software 
Year Produced 2012 
Open Source License? Yes  
Impact PyOP2 is primarily used as a component of Firedrake. Firedrake has a growing community of users, evidenced to some extent by the papers listed at http://firedrakeproject.org/publications.html. PyOP2 is a separate conceptual abstraction which is potentially usable independently. 
URL http://op2.github.io/PyOP2/
 
Description Firedrake Course at CNRS 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The Firedrake team conducted a training course at CNRS in Paris teaching the Firedrake software stack to practitioners from various French institutions.
Year(s) Of Engagement Activity 2017
URL http://calcul.math.cnrs.fr/spip.php?article274
 
Description Invited lecture, London Mathematical Society, Computer Science Day, theme High Performance Scientific Computing at the Exascale. Title: Software Abstractions for many-core software engineering (London, October 2011). 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Paul Kelly was invited to give a public lecture as part of the London Mathematical Society's "Computer Science Day", open to the general public and attended by a mixture of professional practitioners in diverse sectors, as well as researchers in academia and industry.
Year(s) Of Engagement Activity 2011
 
Description Invited talk: ISC15 (session on Programming Models on the Road to Exascale): Domain-Specific Representations in Code Generation for Mesh-Based Computational Science Applications (July 2015) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Paul Kelly was invited to give a short talk as part of a session on software tools on the pathway to exascale computing, at the premier European conference for the HPC/Supercomputing sector. The audience included academic researchers and a strong contingent of industry specialists.
Year(s) Of Engagement Activity 2015
URL http://www.isc-events.com/isc15_ap/sessiondetails.htm?t=session&o=232&a=select&ra=index
 
Description Invited talk: National Science Foundation (NSF) Workshop on High-Level Programming Models for Parallelism.  Title: DSLs for Parallelism:news from the messy frontier -- unstructured-mesh CFD. (Washington, D.C. July 2013). 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Paul Kelly was invited to participate in this event aiming to develop research funding policy for the US National Science Foundation.
Year(s) Of Engagement Activity 2013
 
Description Keynote talk, FEniCS'12: High-performance PDE frameworks for modern architectures.  Title: Generating programs work better than transforming them, if you get the abstraction right (Oslo, June 2012). 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Paul Kelly was invited to present this talk at the workshop for the FEniCS community, primarily scientists working in modeling physical and environmental phenomena and developers of the software that they use.
Year(s) Of Engagement Activity 2012
 
Description Keynote talk: Imperial College HPC Summer School 2016: What your compiler can do, what it will do, and what you might hope it could do 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact This keynote talk formed part of Imperial's HPC Summer School, which aims to build the HPC community and spread best practice among researchers across the whole College. The audience of 70-90 people consisted primarily of researchers at Imperial across all disciplines.
Year(s) Of Engagement Activity 2016
URL http://www.imperial.ac.uk/computational-methods/news-and-events/hpc-2016/
 
Description Keynote talk: LCPC'15 (28th International Workshop on Languages and Compilers for Parallel Computing, Raleigh, North Carolina): Synthesis versus analysis: what do we actually gain from domain-specificity? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Paul Kelly was invited to give the opening keynote at this international workshop/conference. The audience consisted of researchers from industry, academia and national labs.
Year(s) Of Engagement Activity 2015
URL https://www.csc2.ncsu.edu/workshops/lcpc2015/keynote_PaulKelly.html
 
Description Keynote talk: International Workshop on Polyhedral Compilation Techniques (IMPACT) workshop in Stockholm (at the HiPEAC conference): Delivering and generalising domain-specific program optimisations 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Paul Kelly was invited to give the keynote talk at this international research meeting of about 40 people. Participants were researchers from all over the world including PhD students and industrial researchers.
Year(s) Of Engagement Activity 2017
URL http://impact.gforge.inria.fr/impact2017/