Multi-layered abstractions for PDEs
Lead Research Organisation:
Imperial College London
Department Name: Computing
Abstract
Partial differential equations (PDEs) are the mathematical basis ofvast areas of science. From fluid problems such as flow past anaircraft or the circulation of the worlds oceans to structuralproblems in bridges or inside the bones of a body and even in theworld of financial options pricing, PDEs are the key description. Itis therefore unsurprising that computing solutions to PDEs consumes amassive proportion of high performance computing resources in bothindustry and academia.Hitherto, the software which solves PDEs has been written for specificsorts of hardware by people who had to be experts both in the sciencethey were modelling and in programming. This is an expensive anderror-prone process and, because it occurs in an informal way inengineering and science departments, has suffered from a lack of inputfrom modern computer science. However, what makes this proposalcritical for computational science are the large changes which areoccuring in hardware technology. The emergence of new massivelyparallel platforms, such as graphical processing units (GPUs) andmulticore CPU systems is requiring very different approaches tolow-level programming and, importantly, different approaches fordifferent hardware platforms. It also requires new research into thebest optimisation strategies for these platforms. The current approachto writing scientific software will not deliver the changes that arerequired within the industrial and government funding resources available.This project will break this paradigm by using multiple layers ofabstraction to present to scientists and engineers a way ofprogramming PDE problems which is independent of the hardware and tocomputer scientists, a program generation space in which they cangenerate optimal algorithms for PDEs on different hardware platformswithout becoming embroiled in domain science. In particular, we willdevelop a hardware abstraction layer (OP2) which will allow programsto be written by explicitly stating what is parallel. The OP2 layersoftware will then translate this into low-level code which utilisesthe parallelism in a particular hardware platform. For developersworking with the finite element method (FEM), an importantcomputational technique for PDEs, we will present an additional layerwhich will enable them to write the finite element problem in a highlevel language, UFL. This will then be translated to OP2 and thencedown to the hardware layer.We will demonstrate the effectiveness of this approach with two PhDprojects. One will look into the dispersion of polutants in urbansettings while the other will study the stress properties of surgicalbone implants. These are both important scientific questions whichcurrently suffer from a lack of resolution and therefore need to beable to employ the latest high performance hardware.
Planned Impact
This is a high-impact proposal with a direct route to applications both in science and in industry. The potential for impact lies in supporting new applications and new levels of detail and accuracy in computational modelling. Although scientific and engineering computation is not a large part of the UK economy, is has disproportionate impact as an enabling capability across many sectors of industry and in academic research. We believe that, if successful, this project will transform the way PDE codes are written - certainly within the finite-element computational fluid dynamics community and potentially beyond. Our strategy to maximise this impact is to begin by working closely with a small number of close collaborators who are already showing a high level of commitment to this agenda. Once we have demonstrated the value of the technology there, we will invite these key stakeholders to present the results of their experience in community engagement events. Early adopters will include Fluidity (a major academic adaptive, unstructured-mesh finite-element CFD code being developed within Imperial's AMCG group, VOX-FE (a voxel-based finite-element code developed specifically for the structural analysis of bone by Hull's MBE group), and HYDRA (the primary CFD at Rolls Royce, and their largest consumer of CPU cycles). Uptake is recognised as a major management risk for the project. Our primary goal is to embed our technology in major industrial and science codes through existing relationships. We plan to work through the Centre for Fluid Mechanics Simulation (CFMS) and the FEniCS user group to reach further afield, by inviting the owners of these codes to report on their experience. Building on these relationships we will organise workshops to engage with the user community at the end of each year (coordinated with software releases), for which a budget allowance has been made. Further details are provided in Section 5 of the Case for Support and in our Impact Plan.
Organisations
People |
ORCID iD |
Paul Kelly (Principal Investigator) | |
David Ham (Co-Investigator) |
Publications
Bercea G
(2016)
A structure-exploiting numbering algorithm for finite elements on extruded meshes, and its performance evaluation in Firedrake
in Geoscientific Model Development
Bertolli C
(2012)
Mesh independent loop fusion for unstructured mesh applications
Bertolli C
(2013)
Languages and Compilers for Parallel Computing
Cantwell C
(2011)
From h to p efficiently: Strategy selection for operator evaluation on hexahedral and tetrahedral elements
in Computers & Fluids
Cantwell C
(2011)
From h to p Efficiently: Selecting the Optimal Spectral/ hp Discretisation in Three Dimensions
in Mathematical Modelling of Natural Phenomena
Collingbourne P
(2014)
Symbolic Crosschecking of Data-Parallel Floating-Point Code
in IEEE Transactions on Software Engineering
Giles M
(2013)
Designing OP2 for GPU architectures
in Journal of Parallel and Distributed Computing
Giles M
(2011)
Performance analysis of the OP2 framework on many-core architectures
in ACM SIGMETRICS Performance Evaluation Review
Description | We have learned a great deal about how to build really powerful software tools to support scientists who use simulation to model physical systems. The key idea is abstraction: the design of the representation of the problem so that analysis, transformation and synthesis of high-performance code is easy. The original idea of the project was to explore this idea with respect to how parallel computation on the mesh is represented. This idea ("PyOP2") has proven exceptionally powerful: we have used it, for example, to demonstrate, for the first time, how to automatically combine different loops over the mesh even when they have very complex mesh-dependent dependencies. We have discovered two other important levels at which new abstractions help simplify, and thus enable, complicated optimisations. One ("COFFEE") focuses on the local computational loop nest at each element of the mesh, and has enabled us to approach theoretically-optimal reductions in the amount of work needed, while also helping to exploit vector instructions in modern processor architectures. The other new abstraction in the hierarchy is more recent, and builds on work funded by this grant: we have discovered powerful new ways to simplify the code generation process. This "two-stage form compiler" yields dramatically simpler implementation than previous work on automating the finite element method, and exposes new optimisations exploiting the structure of complicated forms found in sophisticated physical models. |
Exploitation Route | The fruits of this research have been embodied in a major open-source software project, Firedrake (http://firedrakeproject.org). Firedrake has an extremely active group of developer and users and is in continuous development. In particular, it is becoming central to continuing research projects in weather, climate, tidal renewables and other geophysical simulation projects in both science and engineering. Meanwhile our research publications have carefully identified the properties and the strategies needed for designing abstractions, or intermediate representations as they are called in the compiler community, in ways which are independent of their embodiment in particular software, and thus transferable to other contexts. |
Sectors | Aerospace, Defence and Marine,Energy,Environment,Healthcare |
URL | http://firedrakeproject.org |
Description | Non-academic impacts are currently at a fairly early stage but are in active, continuing and funded development towards applications. (1) Firedrake is being used to develop new discretisations for improved modelling of the dynamics of the atmosphere, feeding into collaborative work with the UK Met Office on the next-generation weather and climate models (funded by the NERC/Met Office GungHo and GungHo phase II projects). (2) Firedrake is being used to develop new modelling technology for multi-scale interactions between sub-metre scale installations and ocean currents over tens of kilometres is crucial for assessing environmental impacts of coastal engineering projects and tidal energy systems, and for optimisation to minimise project costs or maximise profits. This work is supported by EPSRC (EP/M011054/1) in collaboration with major industry players. |
First Year Of Impact | 2015 |
Sector | Aerospace, Defence and Marine,Environment |
Impact Types | Economic |
Description | A new simulation and optimisation platform for marine technology |
Amount | £434,711 (GBP) |
Funding ID | EP/M011054/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 03/2015 |
End | 02/2018 |
Description | Platform: Underpinning Technologies for Finite Element Simulation |
Amount | £1,287,360 (GBP) |
Funding ID | EP/L000407/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 11/2013 |
End | 10/2018 |
Title | Firedrake |
Description | Firedrake is an automated system for the portable solution of partial differential equations using the finite element method (FEM). Firedrake enables users to employ a wide range of discretisations to an infinite variety of PDEs and employ either conventional CPUs or GPUs to obtain the solution. |
Type Of Technology | Software |
Year Produced | 2013 |
Open Source License? | Yes |
Impact | Firedrake has a growing community of users with whom we are working closely. Some of the resulting papers are listed at http://firedrakeproject.org/publications.html. |
URL | http://www.firedrakeproject.org |
Title | OP2 |
Description | OP2 is framework for the execution of unstructured grid applications on clusters of GPUs or multi-core CPUs. Although OP2 is designed to look like a conventional library, the implementation uses source-source translation to generate the appropriate back-end code for the different target platforms. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | OP2 is being adopted by Rolls Royce to support GPU and multicore acceleration of their flagship HYDRA fluid dynamics software. PyOP2, which is developed from OP2's concepts, is a key abstraction in the Firedrake Project (www.firedrakeproject.org). |
URL | http://www.oerc.ox.ac.uk/projects/op2 |
Title | PyOP2 |
Description | PyOP2 is a domain-specific language (DSL) for the parallel executions of computational kernels on unstructured meshes or graphs. PyOP2 is a key enabling abstraction on which the Firedrake Project is built. It is based on concepts from OP2 (https://github.com/OP2/OP2-Common, http://www.oerc.ox.ac.uk/projects/op2) but is implement in Python and entirely at runtime, rather than as a source-to-source compiler-time tool. It includes substantial extensions and optimisations, mainly to support finite element applications. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | PyOP2 is primarily used as a component of Firedrake. Firedrake has a growing community of users, evidenced to some extent by the papers listed at http://firedrakeproject.org/publications.html. PyOP2 is a separate conceptual abstraction which is potentially usable independently. |
URL | http://op2.github.io/PyOP2/ |
Description | Firedrake Course at CNRS |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The Firedrake team conducted a training course at CNRS in Paris teaching the Firedrake software stack to practitioners from various French institutions. |
Year(s) Of Engagement Activity | 2017 |
URL | http://calcul.math.cnrs.fr/spip.php?article274 |
Description | Invited lecture, London Mathematical Society, Computer Science Day, theme High Performance Scientific Computing at the Exascale. Title: Software Abstractions for many-core software engineering (London, October 2011). |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Paul Kelly was invited to give a public lecture as part of the London Mathematical Society's "Computer Science Day", open to the general public and attended by a mixture of professional practitioners in diverse sectors, as well as researchers in academia and industry. |
Year(s) Of Engagement Activity | 2011 |
Description | Invited talk: ISC15 (session on Programming Models on the Road to Exascale): Domain-Specific Representations in Code Generation for Mesh-Based Computational Science Applications (July 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Paul Kelly was invited to give a short talk as part of a session on software tools on the pathway to exascale computing, at the premier European conference for the HPC/Supercomputing sector. The audience included academic researchers and a strong contingent of industry specialists. |
Year(s) Of Engagement Activity | 2015 |
URL | http://www.isc-events.com/isc15_ap/sessiondetails.htm?t=session&o=232&a=select&ra=index |
Description | Invited talk: National Science Foundation (NSF) Workshop on High-Level Programming Models for Parallelism. Title: DSLs for Parallelism:news from the messy frontier -- unstructured-mesh CFD. (Washington, D.C. July 2013). |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Paul Kelly was invited to participate in this event aiming to develop research funding policy for the US National Science Foundation. |
Year(s) Of Engagement Activity | 2013 |
Description | Keynote talk, FEniCS'12: High-performance PDE frameworks for modern architectures. Title: Generating programs work better than transforming them, if you get the abstraction right (Oslo, June 2012). |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Kelly was invited to present this talk at the workshop for the FEniCS community, primarily scientists working in modeling physical and environmental phenomena and developers of the software that they use. |
Year(s) Of Engagement Activity | 2012 |
Description | Keynote talk: Imperial College HPC Summer School 2016: What your compiler can do, what it will do, and what you might hope it could do |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | This keynote talk formed part of Imperial's HPC Summer School, which aims to build the HPC community and spread best practice among researchers across the whole College. The audience of 70-90 people consisted primarily of researchers at Imperial across all disciplines. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.imperial.ac.uk/computational-methods/news-and-events/hpc-2016/ |
Description | Keynote talk: LCPC'15 (28th International Workshop on Languages and Compilers for Parallel Computing, Raleigh, North Carolina): Synthesis versus analysis: what do we actually gain from domain-specificity? |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Kelly was invited to give the opening keynote at this international workshop/conference. The audience consisted of researchers from industry, academia and national labs. |
Year(s) Of Engagement Activity | 2015 |
URL | https://www.csc2.ncsu.edu/workshops/lcpc2015/keynote_PaulKelly.html |
Description | Keynote talk: International Workshop on Polyhedral Compilation Techniques (IMPACT) workshop in Stockholm (at the HiPEAC conference): Delivering and generalising domain-specific program optimisations |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Kelly was invited to give the keynote talk at this international research meeting of about 40 people. Participants were researchers from all over the world including PhD students and industrial researchers. |
Year(s) Of Engagement Activity | 2017 |
URL | http://impact.gforge.inria.fr/impact2017/ |