Expressive Finite Element Modelling for HPC: enabling advanced techniques for scientists
Lead Research Organisation:
University of Cambridge
Department Name: BP Institute
Abstract
Many aspects of physics can be described by mathematical equations, for example, the motion of water in the sea, the expansion and contraction of metal as it is heated and cooled, or the flow of electricity in a radio antenna.
Scientists write these equations using mathematical symbols. However, when we wish to model a specific problem with a computer, for example the thermal expansion of a particular engine part, there are several practical things which we need to do.
Firstly, we need to be able to describe the shape of the engine part accurately. Usually, the shape can be described by a simple mesh (for example, any two-dimensional shape can be divided into triangles, and three-dimensional shapes can be divided into triangular pyramids, known as tetrahedra). By dividing the complex shape into smaller, simpler shapes the equations can be solved more easily.
Secondly, and most importantly, the mathematical equations describing the physics need to be converted to a form the computer can understand. In the past, this has been a tedious task for programmers, but in the FEniCS scientific software library (fenicsproject.org), this has been made much easier by translating from a mathematical language, similar to the equations, directly into computer code. Along with some additional information about the mesh, and the external conditions (e.g. in the example above, the source of heat), a full model can be computed and the results can be plotted and analysed.
Not all simulations are so straightforward. For example, the equations for air bubbles moving through water would contain terms for surface tension. Surface tension is sensitive to the curvature of the mesh surface, but a mesh made of triangles always has a flat, faceted surface. In a simulation, as the bubbles move, the mesh would also need to be distorted to take account of the new position of the bubbles. In another context, for noise emitted from jet engines, the equations may require complex numbers, or be highly non-linear. In both of these cases, the computer code needs to be written to take care of the specific issue. Scientists would like to make computer simulations without having to worry about the complexities of programming these details.
In this project, I am addressing five main issues: complex numbers, curved surfaces, difficult non-linear problems, mesh deformation, and mesh formats. The FEniCS software framework will be extended to allow users to have access to these features without having to program them directly themselves. It will open up new areas of research, for example in surface tension and liquid-gas interaction, multiphase flow, acoustics, quantum mechanics, and stability analysis, as well as making the software more accessible on HPC platforms.
Software developments alone are not enough to drive forward scientific and engineering advances. Another important aspect of this project is to engage with application scientists and help them to get the most out of software libraries, understand their scientific problem, and help them translate it into computer code. I will provide training, in association with EPSRC doctoral training centres, to use the new software developments at large scale, including on supercomputers. Software maintenance is another important aspect of any scientific project, and I will encourage the next generation of scientists to use modern techniques of version control and automatic testing, which will enhance the quality of their software projects, and improve their longevity.
Scientists write these equations using mathematical symbols. However, when we wish to model a specific problem with a computer, for example the thermal expansion of a particular engine part, there are several practical things which we need to do.
Firstly, we need to be able to describe the shape of the engine part accurately. Usually, the shape can be described by a simple mesh (for example, any two-dimensional shape can be divided into triangles, and three-dimensional shapes can be divided into triangular pyramids, known as tetrahedra). By dividing the complex shape into smaller, simpler shapes the equations can be solved more easily.
Secondly, and most importantly, the mathematical equations describing the physics need to be converted to a form the computer can understand. In the past, this has been a tedious task for programmers, but in the FEniCS scientific software library (fenicsproject.org), this has been made much easier by translating from a mathematical language, similar to the equations, directly into computer code. Along with some additional information about the mesh, and the external conditions (e.g. in the example above, the source of heat), a full model can be computed and the results can be plotted and analysed.
Not all simulations are so straightforward. For example, the equations for air bubbles moving through water would contain terms for surface tension. Surface tension is sensitive to the curvature of the mesh surface, but a mesh made of triangles always has a flat, faceted surface. In a simulation, as the bubbles move, the mesh would also need to be distorted to take account of the new position of the bubbles. In another context, for noise emitted from jet engines, the equations may require complex numbers, or be highly non-linear. In both of these cases, the computer code needs to be written to take care of the specific issue. Scientists would like to make computer simulations without having to worry about the complexities of programming these details.
In this project, I am addressing five main issues: complex numbers, curved surfaces, difficult non-linear problems, mesh deformation, and mesh formats. The FEniCS software framework will be extended to allow users to have access to these features without having to program them directly themselves. It will open up new areas of research, for example in surface tension and liquid-gas interaction, multiphase flow, acoustics, quantum mechanics, and stability analysis, as well as making the software more accessible on HPC platforms.
Software developments alone are not enough to drive forward scientific and engineering advances. Another important aspect of this project is to engage with application scientists and help them to get the most out of software libraries, understand their scientific problem, and help them translate it into computer code. I will provide training, in association with EPSRC doctoral training centres, to use the new software developments at large scale, including on supercomputers. Software maintenance is another important aspect of any scientific project, and I will encourage the next generation of scientists to use modern techniques of version control and automatic testing, which will enhance the quality of their software projects, and improve their longevity.
Planned Impact
The developments of this project have the potential to impact in many areas, including industry, healthcare and society both in the UK and internationally.
The software development proposed will enhance the existing highly popular FEniCS finite element package.
FEniCS has been used for the modelling of: patient-specific physiology, tidal turbine array optimisation, permafrost, glacial flows, the float glass process, protein interaction in solvents, turbulent flow, micro-magnetics, damage and fracture mechanics, strength-weight optimisation, elastic deformation, magma flow, oil reservoirs, carbon capture and storage (CCS) and many other problems. The new developments will enable advances in acoustics, surface tension, moving interfaces, and complex non-linear problems.
Some of these applications have direct industrial impact, as they can be used to model real industrial processes, for example the float glass process, acoustic noise from a jet engine, surface tension in paint droplets, elastic deformation of a rotor blade, or flow in an oil reservoir. By better understanding the way these systems behave, companies can improve their efficiency, and make financial or environmental savings. Direct applications in healthcare include the modelling of patient-specific heart valves, aneurysms or spinal columns, which can help clinicians decide on the best course of treatment for a patient. Finally, some applications have implications for policy, such as the feasibility of CCS storage schemes.
In addition to the software library developments, this project extends into providing a service to scientists and engineers, helping them to produce more robust software, more quickly, and scale up to supercomputers with ease. For industry, this reduces the time from design to obtaining results, and gives the benefit of a more responsive design cycle. Scientists who are supported to use the latest computational tools will also be able to solve problems which they previously thought too large or difficult. This can have a huge impact. Many scientific models are in 2D, simply because the scientists cannot scale up their computations to 3D. With the right support, more accurate modelling in 3D is possible. Larger, more accurate full body medical models for patients may follow, and industrial designs with billions of degrees of freedom which can be solved in minutes.
The software development proposed will enhance the existing highly popular FEniCS finite element package.
FEniCS has been used for the modelling of: patient-specific physiology, tidal turbine array optimisation, permafrost, glacial flows, the float glass process, protein interaction in solvents, turbulent flow, micro-magnetics, damage and fracture mechanics, strength-weight optimisation, elastic deformation, magma flow, oil reservoirs, carbon capture and storage (CCS) and many other problems. The new developments will enable advances in acoustics, surface tension, moving interfaces, and complex non-linear problems.
Some of these applications have direct industrial impact, as they can be used to model real industrial processes, for example the float glass process, acoustic noise from a jet engine, surface tension in paint droplets, elastic deformation of a rotor blade, or flow in an oil reservoir. By better understanding the way these systems behave, companies can improve their efficiency, and make financial or environmental savings. Direct applications in healthcare include the modelling of patient-specific heart valves, aneurysms or spinal columns, which can help clinicians decide on the best course of treatment for a patient. Finally, some applications have implications for policy, such as the feasibility of CCS storage schemes.
In addition to the software library developments, this project extends into providing a service to scientists and engineers, helping them to produce more robust software, more quickly, and scale up to supercomputers with ease. For industry, this reduces the time from design to obtaining results, and gives the benefit of a more responsive design cycle. Scientists who are supported to use the latest computational tools will also be able to solve problems which they previously thought too large or difficult. This can have a huge impact. Many scientific models are in 2D, simply because the scientists cannot scale up their computations to 3D. With the right support, more accurate modelling in 3D is possible. Larger, more accurate full body medical models for patients may follow, and industrial designs with billions of degrees of freedom which can be solved in minutes.
Organisations
- University of Cambridge (Fellow, Lead Research Organisation)
- UNIVERSITY OF EDINBURGH (Collaboration)
- Google (Collaboration)
- University College London (Collaboration)
- University of Luxembourg (Collaboration)
- Technical University of Liberec (Collaboration)
- BP (United States) (Project Partner)
- Melior Innovations (Project Partner)
- University of Oxford (Project Partner)
- Simula Research Laboratory (Project Partner)
- Rice University (Project Partner)
- University of Southampton (Project Partner)
Publications
Shimwell J
(2019)
Multiphysics analysis with CAD-based parametric breeding blanket creation for rapid design iteration
in Nuclear Fusion
Scroggs M
(2022)
Basix: a runtime finite element basis evaluation library
in Journal of Open Source Software
Scroggs M
(2022)
Construction of Arbitrary Order Finite Element Degree-of-Freedom Maps on Polygonal and Polyhedral Cell Meshes
in ACM Transactions on Mathematical Software
Richardson C
(2019)
Scalable computation of thermomechanical turbomachinery problems
in Finite Elements in Analysis and Design
Richardson C
(2019)
Scalable computation of thermomechanical turbomachinery problems
Richardson C
(2018)
Scalable computation of thermomechanical turbomachinery problems
Maljaars J
(2021)
LEoPart: A particle library for FEniCS
in Computers & Mathematics with Applications
Maljaars J
(2019)
LEoPart: a particle library for FEniCS
Hale Jack S.
(2017)
Containers for Portable, Productive, and Performant Scientific Computing
in COMPUTING IN SCIENCE & ENGINEERING
Hale J
(2017)
Containers for Portable, Productive, and Performant Scientific Computing
in Computing in Science & Engineering
Description | The two main outcomes of this award have been software development and community development. Firstly, the technical focus has been on "finite element" software, which has been used extensively for many decades in engineering and the sciences. As part of a collaborative international project, this fellowship added significant new features to an open source (freely available) piece of software, called FEniCS. Secondly, through working with other Research Software Engineers across the UK, and locally, the award helped contribute to a larger and better connected community in this area. |
Exploitation Route | Because the software is so general, widely used, and freely available, it has been taken forward by many researchers across the world, many of which the developers are not fully aware of, and have not directly contributed to. Projects which are a direct outcome include the ASiMoV Prosperity Partnership with Rolls-Royce, and the Excalibur projects on Coupling and SysGenX, all of which are EPSRC funded. |
Sectors | Aerospace Defence and Marine Chemicals Construction Digital/Communication/Information Technologies (including Software) Education Electronics Energy Environment Healthcare |
URL | http://www.fenicsproject.org |
Description | Code contributions to FEniCS Project have been used by Mitsubishi Heavy Industries in a project. Code contributions to FEniCS Project have been used by Rolls Royce in a project. Software outputs have been used in many different projects and subject areas. |
First Year Of Impact | 2018 |
Sector | Aerospace, Defence and Marine,Education,Electronics,Energy,Manufacturing, including Industrial Biotechology,Pharmaceuticals and Medical Biotechnology,Transport |
Impact Types | Economic |
Description | BSS |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Participation in a guidance/advisory committee |
URL | https://betterscientificsoftware.github.io/ |
Description | NumFOCUS Sustainability Forum |
Geographic Reach | North America |
Policy Influence Type | Participation in a guidance/advisory committee |
URL | http://www.numfocus.org |
Description | Participation in RSE-L network |
Geographic Reach | National |
Policy Influence Type | Participation in a guidance/advisory committee |
Description | UK-USA RSE Travel Fund |
Amount | £1,780 (GBP) |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 11/2017 |
End | 03/2018 |
Description | embedded Computational Science & Engineering programme (eCSE) |
Amount | £38,927 (GBP) |
Organisation | University of Edinburgh |
Department | Edinburgh Parallel Computing Centre (EPCC) |
Sector | Academic/University |
Country | United Kingdom |
Start | 03/2017 |
End | 02/2018 |
Description | Containers for Scientific Computation - from laptop to HPC |
Organisation | University of Luxembourg |
Country | Luxembourg |
Sector | Academic/University |
PI Contribution | I performed testing of "docker" container technology on NERSC HPC machines (Edison and Cori) using the shifter implementation. |
Collaborator Contribution | Dr Jack Hale (University of Luxembourg) created the "docker" container material for testing, and we jointly wrote a paper describing the work. |
Impact | Paper accepted in CiSE, awaiting publication: https://arxiv.org/abs/1608.07573 |
Start Year | 2016 |
Description | Erasmus Student hosting |
Organisation | Technical University of Liberec |
Country | Czech Republic |
Sector | Academic/University |
PI Contribution | Hosting and mentoring student from Czech Republic |
Collaborator Contribution | Code development for CI testing on HPC - in collaboration with EPCC and STFC. |
Impact | Report on CI testing from Erasmus Student |
Start Year | 2017 |
Description | Excalibur "ELEMENT" project |
Organisation | University of Edinburgh |
Department | Edinburgh Parallel Computing Centre (EPCC) |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Partner in the Excalibur project |
Collaborator Contribution | Joint workshop and vision paper |
Impact | None yet |
Start Year | 2020 |
Description | Excalibur "SLE" Project |
Organisation | University College London |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Joint Excalibur Project |
Collaborator Contribution | Joint Excalibur Project |
Impact | Workshops resulting in recorded talks, now published on youtube: https://www.youtube.com/channel/UCvWSqJeg1f06X-fieN052Wg |
Start Year | 2020 |
Description | Mentoring of Google Summer of Code student |
Organisation | |
Department | Google Summer of Code |
Country | United States |
Sector | Charity/Non Profit |
PI Contribution | Mentoring and training given to students |
Collaborator Contribution | Code development by students from Slovakia and Finland |
Impact | Code development in the FEniCS Project (fenicsproject.org) |
Start Year | 2017 |
Title | FEniCS |
Description | FEniCS is an open-source finite element library, used for solving scientific and engineering problems. It has been in development for over a decade, with many contributors. |
Type Of Technology | Software |
Year Produced | 2017 |
Open Source License? | Yes |
Impact | The FEniCS book, describing the software has been cited over 800 times since 2012. FEniCS has been used for numerous models in papers from geophysics through to turbomachinery. |
URL | http://fenicsproject.org |
Title | PyROL - Python interface to ROL |
Description | Python interface to ROL Optimisation library from Sandia (USA). |
Type Of Technology | Software |
Year Produced | 2018 |
Open Source License? | Yes |
Impact | Collaboration with Florian Wechsung (Oxford) |
URL | http://bitbucket.org/pyrol/pyrol |
Description | Article for SIAM News |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Article about Rsearch Software Engineering as a new career track |
Year(s) Of Engagement Activity | 2018 |
URL | https://sinews.siam.org/Details-Page/research-software-engineer-a-new-career-track |
Description | CSE17 - Containers for Scientific Computing |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Poster presentation at CSE17 conference in Atlanta, USA. Aroused interest from many researchers about our methodology and results, which will be influential in several institutions across the world. |
Year(s) Of Engagement Activity | 2017 |
URL | https://figshare.com/articles/Containers_for_Scientific_Computing_-_CSE17/4719016 |
Description | Computation Seminar Day |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | Seminar Day for the whole university focussed on Computational Science |
Year(s) Of Engagement Activity | 2018 |
Description | Computational Science and Engineering Conference (Atlanta, USA) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation entitled "The Research Software Engineer- An Emerging New Role in Academia in the UK", discussing the way a new career path is being created in many institutions across the UK. Highly respected academics from across the field of computational science were present, which was a good opportunity to meet them in person. Additionally, several researchers from US and European institutions came to talk to me individually afterwards. |
Year(s) Of Engagement Activity | 2017 |
URL | https://figshare.com/articles/The_Research_Software_Engineer-_An_Emerging_New_Role_in_Academia_in_th... |
Description | Presentation at SIAM-CSE in Spokane |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation about modelling large-scale systems and collaboration with industry |
Year(s) Of Engagement Activity | 2019 |
URL | https://figshare.com/articles/Changing_Mindsets_for_Large-scale_Modelling/7772639 |
Description | SIAM-CSE |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Talk at SIAM-CSE conference on "Using SYCL for Finite Element Assembly" |
Year(s) Of Engagement Activity | 2021 |
URL | https://siamcse21.vfairs.com/ |
Description | Talk at FEniCS16 Oslo - "Multiphysics problems with FEniCS and nested matrices" |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Talk at FEniCS'16 workshop in Oslo entitled "Multiphysics problems with FEniCS and nested matrices" |
Year(s) Of Engagement Activity | 2016 |
URL | http://easychair.org/smart-program/FEniCS'16/2016-05-20.html |
Description | University Annual "Computation Seminar Day" |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | Organised "Seminar Day" for Researchers and Programmers across the University |
Year(s) Of Engagement Activity | 2017,2018 |
Description | Visit to Google (San Jose) for Google Summer of Code Mentor Summit |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Visit to Google (San Jose) for Google Summer of Code Mentor Summit, where I met open source practitioners from different projects around the world. |
Year(s) Of Engagement Activity | 2017 |
Description | WCCM-ECCOMAS presentation |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Talk about "Composable kernels" using new JIT technology in FEniCS for static condensation,. |
Year(s) Of Engagement Activity | 2021 |
URL | https://wccm-eccomas2020.org/ |