Sustainable domain-specific software generation tools for extremely parallel particle-based simulations
Lead Research Organisation:
Imperial College London
Department Name: Computing
Abstract
This research project emerged from the EPSRC Sandpit on Extreme Computing that took place in Jan 2010. The sandpit process facilitated discussion between a group of chemists and civil engineers who all use particle-based numerical simulation approaches. The applications motivating these scientists' simulations vary greatly (from quantum mechanics to dam engineering), very different scales are simulated (atoms in some cases, mm sized sand particles in others) and the interactions between the particles are modelled in significantly different ways. However all of the methods have a common basis (they simulate particles and their interactions) and they all require implementation in a high performance computing (HPC) environment. In fact the similarities in the simulation approach mean that there are key common challenges to achieving effective HPC implementations. Recent and likely future developments in HPC are making massively parallel computations viable; consequently there is a real possibility of achieving a step change in the level of complexity and realism in these numerical models with huge impact on industry and society. However, at the same time, the increased complexity of HPC hardware poses a barrier to being able to fully harness this resource. Working with the chemists and engineers, two computer scientists developed a strategy to address these challenges. The proposed approach is to develop a new Particle Science Language (PSL) to act as a bridge between the scientists who are developing particle based models and the computer science required to implement them in a rapidly evolving HPC environment. The scientists and engineers will implement their physical models in this relatively simple PSL and an automatic code generation system then automatically generate optimized code to effectively run the simulations on high performance systems. The research strategy is to rapidly develop the first implementation of the PSL and prototype it on micron to millimetre scale Discrete Element Modelling (DEM) dynamics code. From this early foundation, we will develop the PSL, with the key objectives of demonstrating generality in the breadth of applications within the particulate dynamics domain and portability to multiple HPC architectures. A range of particle based methods (PBM) are currently used to simulate materials in chemistry, engineering, physics and biophysics. The 4 types of PBM considered directly in the proposed are molecular dynamics (MD), the ONETEP quantum mechanics-based program, discrete element modelling (DEM), and smoothed particle hydrodynamics (SPH). The overall research objective is to develop a sustainable tool that will deliver, in the future, cutting edge research applicable to applications ranging from dam engineering to atomistic drug design. Measureable achievements will also be made during the current project. In fact, we anticipate at least 3 physical science firsts in the project lifetime:(1) The first protein drug simulations that describe the entire protein-drug complex and solvent from first principle quantum mechanics.(2) The first rationalization, at an atomic level, of polymer solubility in CO2, with chemical accuracy. (3) Simulation of the vibrational spectrum of a carbon nanotube from first principles quantum mechanics.(4) Simulation of triaxial compression tests with 1,000,000 particles From a Computer Science perspective the project will lead to a novel high-level compiler optimisation framework that captures dependence, data distribution and loop nest optimisation in a context of highly-irregular data structures beyond the scope of classical approaches such as the polyhedral model, or even new results in shape analysis and ownership-type abstractions.
Planned Impact
The project aim is to generate a step change in the level of realism that can be included in particle-based computer simulations. By enabling sustainable access to the best HPC hardware more sophisticated models of the particles and their interactions will be possible while simulating more particles for longer timescales. The project will deliver very high impact science across a range of length scales. We will develop a means for the physical scientists to focus on understanding their systems without having to exert effort in highly complex HPC code development. At the same time they will be able to directly exploit the latest computer science software technology research. Over the duration of the project and in the future the PSL will enable high impact basic research with many benefits for a range of industries and for society in general. Increasing realism makes the simulation of industrially relevant processes more relevant. Specific examples as well as the consequent benefits are listed below: 1. Engineering analysts: Both DEM and SPH can simulate understand failures such as erosion or landslides, and are useful developing methods to handle the flow of granular materials, amongst many other applications. 2. Industrial chemists using supercritical carbon dioxide: The understanding gained in simulation will enable carbon dioxide to gain more widespread use as a solvent. There are then health and environmental benefits as well as scope for commercial application. 3. Commercial uses of ab-initio MD: The ONETEP code (to be incorporated in the project) already has a strong commercial user base. There will be a significant increase in the scope to use this code for materials and drug development (e.g. by simulating vibrational spectrum of a carbon nanotube). PROJECT TEAM: This project is a new collaboration and there will be much cross-fertilization of information and skills. The research staff that will be taken on will all develop expertise in their specific area of work , however they will also develop their communication skills and a broader scientific understanding as a consequence of the interdisciplinary nature of the research. ACHIEVING IMPACT: Key to delivering benefits will be achieving the research objectives. This will be ensured by careful management of the project, a focus on maintaining the collaborations and recruitment of top quality staff. Workshops at each host university will bring a broader scientific community to the project. Journal publications and presentations at conferences will serve to disseminate the findings to the research community. Industry will be informed of progress via invitations to workshops, communication to existing industrial collaborators.
Organisations
Publications
Bardsley E
(2014)
Computer Aided Verification
Bertolli C
(2013)
Languages and Compilers for Parallel Computing
Betts A
(2018)
Implementing and Evaluating Candidate-Based Invariant Generation
in IEEE Transactions on Software Engineering
Betts A
(2015)
The Design and Implementation of a Verification Technique for GPU Kernels
in ACM Transactions on Programming Languages and Systems
Betts A
(2012)
GPUVerify
Chong N
(2013)
Barrier invariants a shared state abstraction for the analysis of data-dependent GPU kernels
in ACM SIGPLAN Notices
Chong N
(2013)
Barrier invariants
Collingbourne P
(2014)
Symbolic Crosschecking of Data-Parallel Floating-Point Code
in IEEE Transactions on Software Engineering
Collingbourne P
(2011)
Symbolic crosschecking of floating-point and SIMD code
Gorman G
(2012)
Hybrid OpenMP/MPI Anisotropic Mesh Smoothing
in Procedia Computer Science
Hanley K
(2015)
Particle-scale mechanics of sand crushing in compression and shearing using DEM
in Soils and Foundations
Hanley K
(2013)
Temporal variation of contact networks in granular materials
in Granular Matter
Huang X
(2014)
Exploring the influence of interparticle friction on critical state behaviour using DEM
in International Journal for Numerical and Analytical Methods in Geomechanics
Huang X
(2014)
DEM analysis of the influence of the intermediate stress ratio on the critical-state behaviour of granular materials
in Granular Matter
Huang X
(2014)
Effect of sample size on the response of DEM samples with a realistic grading
in Particuology
Huang X
(2016)
Partition of the contact force network obtained in discrete element simulations of element tests
in Computational Particle Mechanics
HUANG X
(2014)
Discrete-element method analysis of the state parameter
in Géotechnique
Huang X
(2017)
Implementation of rotational resistance models: A critical appraisal
in Particuology
Leung E
(2022)
Social support in schools and related outcomes for LGBTQ youth: a scoping review.
in Discover education
Lidbury C
(2015)
Many-core compiler fuzzing
Lidbury C
(2015)
Many-core compiler fuzzing
in ACM SIGPLAN Notices
Luporini F
(2015)
Cross-Loop Optimization of Arithmetic Intensity for Finite Element Local Assembly
in ACM Transactions on Architecture and Code Optimization
Luporini F
(2014)
COFFEE: an Optimizing Compiler for Finite Element Local Assembly
Russell F
(2015)
Optimised three-dimensional Fourier interpolation: An analysis of techniques and application to a linear-scaling density functional theory code
in Computer Physics Communications
Russell F
(2013)
Optimized code generation for finite element local assembly using symbolic manipulation
in ACM Transactions on Mathematical Software
Shire T
(2016)
Constriction size distributions of granular filters: a numerical study
in Géotechnique
Shire T
(2014)
Fabric and Effective Stress Distribution in Internally Unstable Soils
in Journal of Geotechnical and Geoenvironmental Engineering
Sun T
(2020)
A study of vectorization for matrix-free finite element methods
in The International Journal of High Performance Computing Applications
Unat D
(2017)
Trends in Data Locality Abstractions for HPC Systems
in IEEE Transactions on Parallel and Distributed Systems
Wozniak B
(2016)
GiMMiK-Generating bespoke matrix multiplication kernels for accelerators: Application to high-order Computational Fluid Dynamics
in Computer Physics Communications
Zhao X
(2013)
A hybrid smoothed extended finite element/level set method for modeling equilibrium shapes of nano-inhomogeneities
in Computational Mechanics
Description | (1) We have developed a domain-specific performance optimisation applicable to a number of computational science simulations in materials science and chemistry. Our research (embodied in the TINTL library) delivers a substantial performance improvement for the Fourier interpolation step that is common to many Density Functional Theory (DFT) methods. In particular we demonstrated its value and applicability in the widely-used ONETEP package. (2) We made significant progress in enabling large-scale parallel modelling of granular materials (such as sand). In particular we developed (for the first time) systematic analysis of the validity of particular inelastic contact models, and their implementation in the widely-used open-source LAMMPS package. (3) We became involved in efforts to support manycore/GPU programming using formal verification tools, and participated in development of novel techniques for reasoning about parallel programs. The specific research focus was in identifying invariants properties in parallel programs with barrier synchronisations. This research fed into the GPUVerify tool, which has been incorporated into ARM's OpenCL software suite. |
Exploitation Route | Integration of particles into Firedrake is on our Firedrake roadmap; we are exploring how to represent mixed Eulerian/Lagrangian models in Firedrake's DSL. Further exploration of cross-component optimisation opportunities in DFT requires significant further research, and depends on recruitment and training of exceptionally capable interdisciplinary researchers. The GPUVerify work is being pursued energetically by colleagues at Imperial (Alastair Donaldson). |
Sectors | Aerospace, Defence and Marine,Chemicals,Construction,Creative Economy,Digital/Communication/Information Technologies (including Software),Environment,Healthcare,Pharmaceuticals and Medical Biotechnology |
Description | One of the PhD students funded by the project contributed to GPUVerify, a tool for formal verification of correctness properties of GPU kernels. GPUVerify is an open-source software project which has been incorporated into version 2.1 of ARM's Mali Graphics Debugger, a key component in ARM plc's software development tool suite. |
First Year Of Impact | 2015 |
Sector | Digital/Communication/Information Technologies (including Software) |
Impact Types | Economic |
Description | A new simulation and optimisation platform for marine technology |
Amount | £434,711 (GBP) |
Funding ID | EP/M011054/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 03/2015 |
End | 02/2018 |
Description | Platform: Underpinning Technologies for Finite Element Simulation |
Amount | £1,287,360 (GBP) |
Funding ID | EP/L000407/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 11/2013 |
End | 10/2018 |
Title | Firedrake |
Description | Firedrake is an automated system for the portable solution of partial differential equations using the finite element method (FEM). Firedrake enables users to employ a wide range of discretisations to an infinite variety of PDEs and employ either conventional CPUs or GPUs to obtain the solution. |
Type Of Technology | Software |
Year Produced | 2013 |
Open Source License? | Yes |
Impact | Firedrake has a growing community of users with whom we are working closely. Some of the resulting papers are listed at http://firedrakeproject.org/publications.html. |
URL | http://www.firedrakeproject.org |
Title | GPU Verify |
Description | GPUVerify is a tool for analysing graphics processing unit (GPU) kernels written in OpenCL and CUDA. It is a static analysis tool, that is it works purely at compile time, without actually running the GPU kernel. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | GPUVerify has been incorporated into ARM's Mali Graphics Debugger software product (version 2.1) (https://community.arm.com/graphics/b/blog/posts/debugging-opencl-applications-with-mali-graphics-debugger-v2-1-and-gpuverify). The GPUVerify project was led by Dr Alastair Donaldson. Some of the work is attributed to the project "Sustainable domain-specific software generation tools for extremely parallel particle-based simulations" due to the involvement of Nathan Chong in the work. |
URL | http://multicore.doc.ic.ac.uk/tools/GPUVerify/ |
Title | OP2 |
Description | OP2 is framework for the execution of unstructured grid applications on clusters of GPUs or multi-core CPUs. Although OP2 is designed to look like a conventional library, the implementation uses source-source translation to generate the appropriate back-end code for the different target platforms. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | OP2 is being adopted by Rolls Royce to support GPU and multicore acceleration of their flagship HYDRA fluid dynamics software. PyOP2, which is developed from OP2's concepts, is a key abstraction in the Firedrake Project (www.firedrakeproject.org). |
URL | http://www.oerc.ox.ac.uk/projects/op2 |
Title | PRAgMaTIc (Parallel anisotRopic Adaptive Mesh ToolkIt) |
Description | PRAgMaTIc (Parallel anisotRopic Adaptive Mesh ToolkIt) provides 2D/3D anisotropic mesh adaptivity for meshes of simplexes. The target applications are finite element and finite volume methods although it can also be used as a lossy compression algorithm for 2 and 3D data (e.g. image compression). It takes as its input the mesh and a metric tensor field which encodes desired mesh element size anisotropically. The toolkit is written in C++ but also provides interfaces for Fortran. It has been integrated with FEniCS/Dolfin and integration with PETSc/DMPlex is planned. One of the design goals of PRAgMaTIc is to develop highly scalable algorithms for clusters of multi-core and many-core nodes. PRAgMaTIc uses OpenMP for thread parallelism and MPI for domain decomposition parallelisation. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | PRAgMaTIc has been integrated into the DMPLEX component of the very widely-used PetSc library, https://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/DMPLEX/DMPlexAdapt.html |
URL | http://meshadaptation.github.io/ |
Title | PyOP2 |
Description | PyOP2 is a domain-specific language (DSL) for the parallel executions of computational kernels on unstructured meshes or graphs. PyOP2 is a key enabling abstraction on which the Firedrake Project is built. It is based on concepts from OP2 (https://github.com/OP2/OP2-Common, http://www.oerc.ox.ac.uk/projects/op2) but is implement in Python and entirely at runtime, rather than as a source-to-source compiler-time tool. It includes substantial extensions and optimisations, mainly to support finite element applications. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | PyOP2 is primarily used as a component of Firedrake. Firedrake has a growing community of users, evidenced to some extent by the papers listed at http://firedrakeproject.org/publications.html. PyOP2 is a separate conceptual abstraction which is potentially usable independently. |
URL | http://op2.github.io/PyOP2/ |
Title | TINTL |
Description | TINTL implements Fourier resampling of 3D regular data-sets to twice their resolution. By various algorithmic and implementation techniques, notably avoiding unnecessary computation with zeroes, TINTL achieves a substantial speedup for this operation. |
Type Of Technology | Software |
Year Produced | 2014 |
Open Source License? | Yes |
Impact | Fourier resampling is a major element of the execution time for Density Functional Theory codes. We have integrated TINTL into ONETEP and have achieved compelling performance results. We are currently working with the ONETEP developers to deliver TINTL to ONETEP users. |
URL | https://github.com/FrancisRussell/tintl |
Description | Invited lecture, London Mathematical Society, Computer Science Day, theme High Performance Scientific Computing at the Exascale. Title: Software Abstractions for many-core software engineering (London, October 2011). |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Paul Kelly was invited to give a public lecture as part of the London Mathematical Society's "Computer Science Day", open to the general public and attended by a mixture of professional practitioners in diverse sectors, as well as researchers in academia and industry. |
Year(s) Of Engagement Activity | 2011 |
Description | Invited talk: ISC15 (session on Programming Models on the Road to Exascale): Domain-Specific Representations in Code Generation for Mesh-Based Computational Science Applications (July 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Paul Kelly was invited to give a short talk as part of a session on software tools on the pathway to exascale computing, at the premier European conference for the HPC/Supercomputing sector. The audience included academic researchers and a strong contingent of industry specialists. |
Year(s) Of Engagement Activity | 2015 |
URL | http://www.isc-events.com/isc15_ap/sessiondetails.htm?t=session&o=232&a=select&ra=index |
Description | Invited talk: National Science Foundation (NSF) Workshop on High-Level Programming Models for Parallelism. Title: DSLs for Parallelism:news from the messy frontier -- unstructured-mesh CFD. (Washington, D.C. July 2013). |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Paul Kelly was invited to participate in this event aiming to develop research funding policy for the US National Science Foundation. |
Year(s) Of Engagement Activity | 2013 |
Description | Keynote talk, FEniCS'12: High-performance PDE frameworks for modern architectures. Title: Generating programs work better than transforming them, if you get the abstraction right (Oslo, June 2012). |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Kelly was invited to present this talk at the workshop for the FEniCS community, primarily scientists working in modeling physical and environmental phenomena and developers of the software that they use. |
Year(s) Of Engagement Activity | 2012 |
Description | Keynote talk: Imperial College HPC Summer School 2016: What your compiler can do, what it will do, and what you might hope it could do |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | This keynote talk formed part of Imperial's HPC Summer School, which aims to build the HPC community and spread best practice among researchers across the whole College. The audience of 70-90 people consisted primarily of researchers at Imperial across all disciplines. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.imperial.ac.uk/computational-methods/news-and-events/hpc-2016/ |
Description | Keynote talk: LCPC'15 (28th International Workshop on Languages and Compilers for Parallel Computing, Raleigh, North Carolina): Synthesis versus analysis: what do we actually gain from domain-specificity? |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Kelly was invited to give the opening keynote at this international workshop/conference. The audience consisted of researchers from industry, academia and national labs. |
Year(s) Of Engagement Activity | 2015 |
URL | https://www.csc2.ncsu.edu/workshops/lcpc2015/keynote_PaulKelly.html |
Description | Keynote talk: International Workshop on Polyhedral Compilation Techniques (IMPACT) workshop in Stockholm (at the HiPEAC conference): Delivering and generalising domain-specific program optimisations |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Kelly was invited to give the keynote talk at this international research meeting of about 40 people. Participants were researchers from all over the world including PhD students and industrial researchers. |
Year(s) Of Engagement Activity | 2017 |
URL | http://impact.gforge.inria.fr/impact2017/ |