Expressive Finite Element Modelling for HPC: enabling advanced techniques for scientists

Lead Research Organisation: University of Cambridge
Department Name: BP Institute

Abstract

Many aspects of physics can be described by mathematical equations, for example, the motion of water in the sea, the expansion and contraction of metal as it is heated and cooled, or the flow of electricity in a radio antenna.

Scientists write these equations using mathematical symbols. However, when we wish to model a specific problem with a computer, for example the thermal expansion of a particular engine part, there are several practical things which we need to do.
Firstly, we need to be able to describe the shape of the engine part accurately. Usually, the shape can be described by a simple mesh (for example, any two-dimensional shape can be divided into triangles, and three-dimensional shapes can be divided into triangular pyramids, known as tetrahedra). By dividing the complex shape into smaller, simpler shapes the equations can be solved more easily.
Secondly, and most importantly, the mathematical equations describing the physics need to be converted to a form the computer can understand. In the past, this has been a tedious task for programmers, but in the FEniCS scientific software library (fenicsproject.org), this has been made much easier by translating from a mathematical language, similar to the equations, directly into computer code. Along with some additional information about the mesh, and the external conditions (e.g. in the example above, the source of heat), a full model can be computed and the results can be plotted and analysed.

Not all simulations are so straightforward. For example, the equations for air bubbles moving through water would contain terms for surface tension. Surface tension is sensitive to the curvature of the mesh surface, but a mesh made of triangles always has a flat, faceted surface. In a simulation, as the bubbles move, the mesh would also need to be distorted to take account of the new position of the bubbles. In another context, for noise emitted from jet engines, the equations may require complex numbers, or be highly non-linear. In both of these cases, the computer code needs to be written to take care of the specific issue. Scientists would like to make computer simulations without having to worry about the complexities of programming these details.

In this project, I am addressing five main issues: complex numbers, curved surfaces, difficult non-linear problems, mesh deformation, and mesh formats. The FEniCS software framework will be extended to allow users to have access to these features without having to program them directly themselves. It will open up new areas of research, for example in surface tension and liquid-gas interaction, multiphase flow, acoustics, quantum mechanics, and stability analysis, as well as making the software more accessible on HPC platforms.

Software developments alone are not enough to drive forward scientific and engineering advances. Another important aspect of this project is to engage with application scientists and help them to get the most out of software libraries, understand their scientific problem, and help them translate it into computer code. I will provide training, in association with EPSRC doctoral training centres, to use the new software developments at large scale, including on supercomputers. Software maintenance is another important aspect of any scientific project, and I will encourage the next generation of scientists to use modern techniques of version control and automatic testing, which will enhance the quality of their software projects, and improve their longevity.

Planned Impact

The developments of this project have the potential to impact in many areas, including industry, healthcare and society both in the UK and internationally.

The software development proposed will enhance the existing highly popular FEniCS finite element package.
FEniCS has been used for the modelling of: patient-specific physiology, tidal turbine array optimisation, permafrost, glacial flows, the float glass process, protein interaction in solvents, turbulent flow, micro-magnetics, damage and fracture mechanics, strength-weight optimisation, elastic deformation, magma flow, oil reservoirs, carbon capture and storage (CCS) and many other problems. The new developments will enable advances in acoustics, surface tension, moving interfaces, and complex non-linear problems.

Some of these applications have direct industrial impact, as they can be used to model real industrial processes, for example the float glass process, acoustic noise from a jet engine, surface tension in paint droplets, elastic deformation of a rotor blade, or flow in an oil reservoir. By better understanding the way these systems behave, companies can improve their efficiency, and make financial or environmental savings. Direct applications in healthcare include the modelling of patient-specific heart valves, aneurysms or spinal columns, which can help clinicians decide on the best course of treatment for a patient. Finally, some applications have implications for policy, such as the feasibility of CCS storage schemes.

In addition to the software library developments, this project extends into providing a service to scientists and engineers, helping them to produce more robust software, more quickly, and scale up to supercomputers with ease. For industry, this reduces the time from design to obtaining results, and gives the benefit of a more responsive design cycle. Scientists who are supported to use the latest computational tools will also be able to solve problems which they previously thought too large or difficult. This can have a huge impact. Many scientific models are in 2D, simply because the scientists cannot scale up their computations to 3D. With the right support, more accurate modelling in 3D is possible. Larger, more accurate full body medical models for patients may follow, and industrial designs with billions of degrees of freedom which can be solved in minutes.

Publications

10 25 50
publication icon
Daversin-Catty C (2021) Abstractions and Automated Algorithms for Mixed Domain Finite Element Methods in ACM Transactions on Mathematical Software

publication icon
Hale J (2017) Containers for Portable, Productive, and Performant Scientific Computing in Computing in Science & Engineering

publication icon
Hale Jack S. (2017) Containers for Portable, Productive, and Performant Scientific Computing in COMPUTING IN SCIENCE & ENGINEERING

publication icon
Maljaars J (2021) LEoPart: A particle library for FEniCS in Computers & Mathematics with Applications

publication icon
Richardson C (2019) Scalable computation of thermomechanical turbomachinery problems in Finite Elements in Analysis and Design

publication icon
Scroggs M (2022) Basix: a runtime finite element basis evaluation library in Journal of Open Source Software

 
Description The two main outcomes of this award have been software development and community development. Firstly, the technical focus has been on "finite element" software, which has been used extensively for many decades in engineering and the sciences. As part of a collaborative international project, this fellowship added significant new features to an open source (freely available) piece of software, called FEniCS. Secondly, through working with other Research Software Engineers across the UK, and locally, the award helped contribute to a larger and better connected community in this area.
Exploitation Route Because the software is so general, widely used, and freely available, it has been taken forward by many researchers across the world, many of which the developers are not fully aware of, and have not directly contributed to. Projects which are a direct outcome include the ASiMoV Prosperity Partnership with Rolls-Royce, and the Excalibur projects on Coupling and SysGenX, all of which are EPSRC funded.
Sectors Aerospace, Defence and Marine,Chemicals,Construction,Digital/Communication/Information Technologies (including Software),Education,Electronics,Energy,Environment,Healthcare

URL http://www.fenicsproject.org
 
Description Code contributions to FEniCS Project have been used by Mitsubishi Heavy Industries in a project. Code contributions to FEniCS Project have been used by Rolls Royce in a project. Software outputs have been used in many different projects and subject areas.
First Year Of Impact 2018
Sector Aerospace, Defence and Marine,Education,Energy,Manufacturing, including Industrial Biotechology,Pharmaceuticals and Medical Biotechnology
Impact Types Economic

 
Description BSS
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a guidance/advisory committee
URL https://betterscientificsoftware.github.io/
 
Description NumFOCUS Sustainability Forum
Geographic Reach North America 
Policy Influence Type Participation in a guidance/advisory committee
URL http://www.numfocus.org
 
Description Participation in RSE-L network
Geographic Reach National 
Policy Influence Type Participation in a guidance/advisory committee
 
Description UK-USA RSE Travel Fund
Amount £1,780 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 11/2017 
End 03/2018
 
Description embedded Computational Science & Engineering programme (eCSE)
Amount £38,927 (GBP)
Organisation University of Edinburgh 
Department Edinburgh Parallel Computing Centre (EPCC)
Sector Academic/University
Country United Kingdom
Start 03/2017 
End 02/2018
 
Description Containers for Scientific Computation - from laptop to HPC 
Organisation University of Luxembourg
Country Luxembourg 
Sector Academic/University 
PI Contribution I performed testing of "docker" container technology on NERSC HPC machines (Edison and Cori) using the shifter implementation.
Collaborator Contribution Dr Jack Hale (University of Luxembourg) created the "docker" container material for testing, and we jointly wrote a paper describing the work.
Impact Paper accepted in CiSE, awaiting publication: https://arxiv.org/abs/1608.07573
Start Year 2016
 
Description Erasmus Student hosting 
Organisation Technical University of Liberec
Country Czech Republic 
Sector Academic/University 
PI Contribution Hosting and mentoring student from Czech Republic
Collaborator Contribution Code development for CI testing on HPC - in collaboration with EPCC and STFC.
Impact Report on CI testing from Erasmus Student
Start Year 2017
 
Description Excalibur "ELEMENT" project 
Organisation University of Edinburgh
Department Edinburgh Parallel Computing Centre (EPCC)
Country United Kingdom 
Sector Academic/University 
PI Contribution Partner in the Excalibur project
Collaborator Contribution Joint workshop and vision paper
Impact None yet
Start Year 2020
 
Description Excalibur "SLE" Project 
Organisation University College London
Country United Kingdom 
Sector Academic/University 
PI Contribution Joint Excalibur Project
Collaborator Contribution Joint Excalibur Project
Impact Workshops resulting in recorded talks, now published on youtube: https://www.youtube.com/channel/UCvWSqJeg1f06X-fieN052Wg
Start Year 2020
 
Description Mentoring of Google Summer of Code student 
Organisation Google
Department Google Summer of Code
Country United States 
Sector Charity/Non Profit 
PI Contribution Mentoring and training given to students
Collaborator Contribution Code development by students from Slovakia and Finland
Impact Code development in the FEniCS Project (fenicsproject.org)
Start Year 2017
 
Title FEniCS 
Description FEniCS is an open-source finite element library, used for solving scientific and engineering problems. It has been in development for over a decade, with many contributors. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact The FEniCS book, describing the software has been cited over 800 times since 2012. FEniCS has been used for numerous models in papers from geophysics through to turbomachinery. 
URL http://fenicsproject.org
 
Title PyROL - Python interface to ROL 
Description Python interface to ROL Optimisation library from Sandia (USA). 
Type Of Technology Software 
Year Produced 2018 
Open Source License? Yes  
Impact Collaboration with Florian Wechsung (Oxford) 
URL http://bitbucket.org/pyrol/pyrol
 
Description Article for SIAM News 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Article about Rsearch Software Engineering as a new career track
Year(s) Of Engagement Activity 2018
URL https://sinews.siam.org/Details-Page/research-software-engineer-a-new-career-track
 
Description CSE17 - Containers for Scientific Computing 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Poster presentation at CSE17 conference in Atlanta, USA. Aroused interest from many researchers about our methodology and results, which will be influential in several institutions across the world.
Year(s) Of Engagement Activity 2017
URL https://figshare.com/articles/Containers_for_Scientific_Computing_-_CSE17/4719016
 
Description Computation Seminar Day 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact Seminar Day for the whole university focussed on Computational Science
Year(s) Of Engagement Activity 2018
 
Description Computational Science and Engineering Conference (Atlanta, USA) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation entitled "The Research Software Engineer- An Emerging New Role in Academia in the UK", discussing the way a new career path is being created in many institutions across the UK. Highly respected academics from across the field of computational science were present, which was a good opportunity to meet them in person. Additionally, several researchers from US and European institutions came to talk to me individually afterwards.
Year(s) Of Engagement Activity 2017
URL https://figshare.com/articles/The_Research_Software_Engineer-_An_Emerging_New_Role_in_Academia_in_th...
 
Description Presentation at SIAM-CSE in Spokane 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation about modelling large-scale systems and collaboration with industry
Year(s) Of Engagement Activity 2019
URL https://figshare.com/articles/Changing_Mindsets_for_Large-scale_Modelling/7772639
 
Description SIAM-CSE 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Talk at SIAM-CSE conference on "Using SYCL for Finite Element Assembly"
Year(s) Of Engagement Activity 2021
URL https://siamcse21.vfairs.com/
 
Description Talk at FEniCS16 Oslo - "Multiphysics problems with FEniCS and nested matrices" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Talk at FEniCS'16 workshop in Oslo entitled "Multiphysics problems with FEniCS and nested matrices"
Year(s) Of Engagement Activity 2016
URL http://easychair.org/smart-program/FEniCS'16/2016-05-20.html
 
Description University Annual "Computation Seminar Day" 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Organised "Seminar Day" for Researchers and Programmers across the University
Year(s) Of Engagement Activity 2017,2018
 
Description Visit to Google (San Jose) for Google Summer of Code Mentor Summit 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Visit to Google (San Jose) for Google Summer of Code Mentor Summit, where I met open source practitioners from different projects around the world.
Year(s) Of Engagement Activity 2017
 
Description WCCM-ECCOMAS presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Talk about "Composable kernels" using new JIT technology in FEniCS for static condensation,.
Year(s) Of Engagement Activity 2021
URL https://wccm-eccomas2020.org/