Software Engineering - In Support of the Exascale

Lead Research Organisation: University of Oxford
Department Name: Engineering Science

Abstract

The Software Engineer, the forgotten enabler of ... well everything nowadays! Wherever there is a computer there needs to be software, otherwise it is just a lump of silicon, metal and plastic. And wherever software is needed the man or woman you call upon is the Software Engineer. So whether it is your phone, tablet or PC, or your TV, toaster or washing machine, or a host of other appliances nowadays the Software Engineer is the person who makes it work. And good software engineering is hard! "Computer says no" and "Blue Screen of Death" are just two memes that indicate the frustration caused in modern life by software that does not behave as desired, and in turn demonstrates the need for quality software to make life, at the very least, just that little bit more bearable.

The Research Software Engineer is what it says, the Software Engineer operating in a research environment. These are the people who provide the crucial infrastructure that supports and enables much of modern research, for whether it be in the arts and humanities or the physical sciences or anywhere in between the computer is the crucial modern tool. So this funding is to support one such guy who, while not producing reams of papers himself, enables literally thousands of other researchers to do their work. In particular I work in High Performance Computing (HPC), possibly better known as supercomputing. Through this grant I'm going to develop a number of scientific applications so that they can not only use more effectively modern top end computing facilities, but also I will provide new capabilities within those programs so that brand new quantities can be calculated. So, for instance, modern supercomputers consist of tens or hundreds of thousands of individual computers all working together. Now we know from life how difficult it is to get a small group of people to all pull together to get something done at all, let alone well, my job is to achieve this with hundreds of thousands of workers all trying to solve a complex scientific problem in the most efficient way possible while getting the right answer. It's not easy, but that's what makes it fun!

The changes I will implement will benefit a very large number of researchers, the applications that I plan to work on (DL_POLY_4, CRYSTAL, CRYSCOR and GS2) have around 3,500 licensed groups in total and that number is growing daily. The subject areas enabled range from the modelling the microscopic, quantum described, structure of materials through looking at the dynamics of large macromolecules, right up to modelling fusion generation of power. Thus just one ultimate beneficiary is the ITER project, a 10 billion Euro experimental tokamak being built in France, and other areas include catalysts, pharmaceuticals, sensors, fuel cells, electronic devices, batteries and a host of others limited only by the researcher's imagination.

But it's not just coding! I've been in this game quite a while now and want to hand on my experiences to the new generation. It's becoming increasingly clear that with the more and more complex computational tools that are being used in research that there is a real skills gap in software, leaving that wonderful piece of hardware as just, well a pile of silicon, metal and plastic. So part of the funding is to train and produce training material for young graduate students who really need expertise in this area, but the opportunities to learn certain key aspects are missing.

So overall this is funding to provide the infrastructure to push at least part of the HPC landscape towards the 2020s, and with a bit of luck beyond!

Planned Impact

The project will deliver impact to academic beneficiaries in conventional ways, including papers in both journals and conferences. However its main outputs will be software developments in a number of widely used software applications, DL_POLY, CRYSTAL, CRYSCOR and GS2, which between them have almost 3,5000 licensed groups worldwide. and the training of young researchers, primarily at CDTs. The managed software releases, as described in the Pathways to Impact document, will be available free to all UK academic researchers and will have gone through the standard software quality checks for each fo the applications in question. The training material will not only be physically delivers, it will also be made free available under a non-commercial creative commons licence. The fellowship will thus benefit a very large number of scientists in a wide variety of career stages by providing computational capabilities not found elsewhere and explaining how to use it. It will further benefit the authors of the codes, providing enhanced capabilities and scalability for their products. It will also benefit academics by raising the profile of the Research Software Engineer, thus supporting a culture of quality, sustainable software within academic research.

Outside the immediate HPC community the code developments will in time trickle down from the top end machines to those more commonly encountered through the evolution of both the hardware and software available to researchers. Similarly the training of students in the elements of HPC will benefit not only those that go onto use HPC in their day to day work environment as the insight it gives to how machines really work and the greater sympathy for how a software and hardware interact results in more efficient and usable software at all levels of computing.

In the wider community the number of potential beneficiaries is immense. The materials science applications (CRYSTAL, CRYSCOR, DL_POLY) all have obvious commercial application in a wide variety of areas, a very partial list being the chemical and pharmaceutical industries, catalysis, energy storage methodologies, solar cells, electronic devices, lasers and laser therapy, and high strength materials. Not unrelated to the exploitation of the new code in research and industry will be the awareness that this work was done in the UK, thus raising awareness of the capability within the nation in this area.

Finally I return to the point that the mere existence of an RSE post funded by a research council is a major step forward. This will not only raise the profile of a crucial part of the research infrastructure of the nation, it will also make younger researchers aware that there is a viable career path as an RSE, thus strengthening UK, and world, research, over the coming years.

Publications

10 25 50
 
Description Member of ARCHER2 PWG
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
 
Description Membership of the Archer2 Project Evaluation Team
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
 
Description Archer Instant Access Award
Amount £1,000 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Department ARCHER Service
Sector Academic/University
Country United Kingdom
Start 10/2017 
End 03/2018
 
Description Archer RAP Award
Amount £34,062 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Department ARCHER Service
Sector Academic/University
Country United Kingdom
Start 12/2017 
End 12/2018
 
Title CRYSTAL 2017 
Description The CRYSTAL package performs ab initio calculations of the ground state energy, energy gradient, electronic wave function and properties of periodic systems. Hartree-Fock or Kohn-Sham Hamiltonians (that adopt an Exchange-Correlation potential following the postulates of Density-Functional Theory) can be used. Systems periodic in 0 (molecules, 0D), 1 (polymers, 1D), 2 (slabs, 2D), and 3 dimensions (crystals, 3D) are treated on an equal footing. The code employs a local basis set, namely linear combinations of Gaussian type functions (GTF). The program can automatically handle space symmetry: 
Type Of Technology Software 
Year Produced 2017 
Impact This is a new release of the long standing CRYSTAL code which is licensed to over 500 groups world wide. This release includes a number of new features, and in particular its parallel capabilities and capcity to successfully handle extremely large system sizes caused news of this release to form the front cover of the October 2017 edition of The Journal of Chemical Theory and Computation. 
URL http://www.crystal.unito.it/index.php
 
Title QuEST - The Quantum Exact Simulation Toolkit 
Description The Quantum Exact Simulation Toolkit is a high performance simulator of universal quantum circuits. QuEST is written in C, hybridises OpenMP and MPI, and can run on a GPU. Needing only compilation, QuEST is easy to run both on laptops and supercomputers, where it can take advantage of multicore and networked machines to quickly simulate circuits on many qubits. This work is in collaboration with Professor Simon Benjamins group (Oxford), part of the NQIT consortium 
Type Of Technology Software 
Year Produced 2018 
Open Source License? Yes  
Impact This is the initial release of the code. It is the only open source simulator for quantum circuits that supports all of shared memory, distributed memory and GPU parallel programming environments. Three groups in Oxford have started to use the package, and in a paper submitted to Quantum we have shown that the capabilities are at least as good as existing simulators where comparison can be made - for the distributed memory case this is difficult as QuEST is the only open source package that supports this, and given the huge memory requirements of simulating Quantum circuits (For N quantum bits 2^N classical bytes are required, and thus for N greater than about 35 machines such as ARCHER are required) this is a extremely important feature. 
URL https://github.com/aniabrown/QuEST
 
Title QuEST - The Quantum Exact Simulation Toolkit - Further Development 
Description The Quantum Exact Simulation Toolkit is a high performance simulator of universal quantum circuits. QuEST is written in C, hybridises OpenMP and MPI, and can run on a GPU. Needing only compilation, QuEST is easy to run both on laptops and supercomputers, where it can take advantage of multicore and networked machines to quickly simulate circuits on many qubits. This work is in collaboration with Professor Simon Benjamins group (Oxford), part of the NQIT consortium 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact After last years initial release, this year saw versions v1.0.0 (10/05/2018), v1.1.0 (14/07/2018), v2.0.0 (12/12/2018), v2.0.1 (20/02/2019) being made available. These have increased functionality, such as dealing with density matrices, and improved performance. Pull requests for the software are now being made from a number of groups. 
URL https://quest.qtechtheory.org/
 
Description "Profiling large scale MPI programs with Extrae and Paraver" -- Bout++ workshop 2018, York, UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We reported our results of large scaling profiling runs on the code GS2 to the authors of a related code, BOUT++. Interest was shown as how to apply our methods to the new code
Year(s) Of Engagement Activity 2018
URL https://boutproject.github.io/documentation/workshop2018.html
 
Description "QuEST and the importance of the interconnect in large scale quantum circuit simulation" - Archer Champions 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact We presented the results of our work with QuEST on Archer and on the local Oxford cluster Arcus, comparing and contrasting the results. The Q&A session afterwards and interaction with a number of hardware engineers from Cray afterwards helped us rationalise our result, and will contribute to a publication
Year(s) Of Engagement Activity 2018
URL https://www.archer.ac.uk/community/champions/workshops/manchester_apr2018/
 
Description "RSE and RSE's in the UK - A Recent History and a Personal Viewpoint" - talk at All Hand Meeting, NAG, Oxford, April 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A talk on the position of Research Software Engineers within the UK academic research environment was given at the Numerical Algorithms All Hands Meeting, as well as presenting some of my own and my groups work as an RSE. This sparked debate on how NAG and the software industry more generally can interact with the growing numbers of RSE and RSE groups in UK Universities.
Year(s) Of Engagement Activity 2018
 
Description Archer Champions Meeting 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact On June 26-27th, Anna travelled to Daresbury for an Archer Champions meeting. This is organised by EPCC for the purpose of keeping 'Archer Champions', who perform outreach work related to the Archer national supercomputer, up to date on the state of the Archer system. The meeting included updates on several newly developed 'Tier-2' supercomputing systems and examples of outreach work done by the EPCC team, which we hope to build on in our local area.
Year(s) Of Engagement Activity 2017
URL http://www.archer.ac.uk/community/champions/
 
Description CRYSTAL Developers Meeting 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A meeting for the primary developers of the CRYSTAL ab initio electronic structure code was held at the University of Oxford. Future development plans were made, as were plans for the future distribution of the code, and educationof its users
Year(s) Of Engagement Activity 2019
 
Description Debugging Numerical Software, Bath, UK (4/06/2018) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact This workshop gathered together practitioners of programming numerical software with a particular interest the use of debuggers, and teaching their use. A practical example was provided, and useful techniques were leant from other delegates at the workshop.
Year(s) Of Engagement Activity 2018
URL https://sites.google.com/site/debuggingnumericalsoftware/
 
Description HPC Workshop - Biomolecular Simulations, Machine Learning and AI at Oxford 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact London Economics on behalf of EPSRC organised a number of workshops to discuss and round out future HPC policy in the UK, especially with regard to funding and provision.
Year(s) Of Engagement Activity 2019
 
Description KNL Meeting - Bologna 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact On March 22-23rd, Anna attended a workshop on the Xeon Phi (KNL) architecture, organised by Lenovo. The purpose was to better evaluate the KNL architecture (a promising but unproven high performance device) and its relevance to our work, as well as to receive optimisation training.
Year(s) Of Engagement Activity 2017
 
Description MSSC 2018 at Imperial College, London 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A talk was given on the use and performance of the CRYSTAL ab initio electronic structure code, focusing on how to use it best on parallel supercomputers. Much interaction was had in subsequent coffee breaks with the attendees on how best to go about solving their particular research problems
Year(s) Of Engagement Activity 2018
URL http://www.imperial.ac.uk/mssc2018
 
Description N8-CIR New Horizons in Materials Modelling 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Work by members of the group was presented, and the audience expressed interest in the materials presented
Year(s) Of Engagement Activity 2019
URL https://n8cir.org.uk/news/new-horizons-materials-modelling/
 
Description Present "An introduction to GPU optimisation using the NVIDIA visual profiler" Workshop at RSE Conference, Manchester 2017 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Graphical processing units (GPUs) are a vital HPC tool due to their high computing power and good energy efficiency compared to CPUs. However it is not straightforward to write performant GPU code due to certain requirements of the architecture, including management of the memory hierarchy, the need for high computational throughput and grouping of thread execution. To help understand these critical optimisation opportunities this workshop introduced the NVIDIA Visual Profiler (NVVP), and through that basic optimisation strategies for GPU application developers. The workshop was aimed at developers with beginning to intermediate level CUDA. Reslting from this workshop plans are being made to develop further, related training material.
Year(s) Of Engagement Activity 2017
URL http://rse.ac.uk/conf2017/programme/
 
Description Present Talk At "CCP9 Community Meeting" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I presented a talk to CCP9 researchers on the role of the RSE in science, the status of the RSE movement in British science, how RSE groups are supporting scientists and enabling new science, and in particular on how my role in such a post had over years resulted in CRYSTAL and DL_POLY being the scalable codes that they are today.
Year(s) Of Engagement Activity 2017
URL https://www.ccp9.ac.uk/node/1470
 
Description Present Talk At "CCP9 Young Researchers Event and Community Meeting 2017" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I presented a talk to young researchers on the role of the RSE in science, and in particular on how my role in such a post had over years resulted in CRYSTAL and DL_POLY being the scalable codes that they are today. There was also a round table session at the end of the day.
Year(s) Of Engagement Activity 2017
URL https://www.ccp9.ac.uk/node/1470
 
Description Present Talk on Fellowship work to the HPC Materials Chemistry Consortium 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact As part of becoming a full member of the consortium I presented a talk outlining the role of the RSE, and describing my work in areas relevant to the consortium
Year(s) Of Engagement Activity 2017
URL http://www.ucl.ac.uk/klmc/mcc/
 
Description Profiling GS2 on large process counts with the paraver profiling tool" -- GS2 Developer meeting, Abingdon, UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We presented our work on the gyrokinetic code GS2 to fellow developers in a meeting bringing together all working on the code. The profiling results and potential optimisation of the code cause much interest, and should in the future save much computer time.
Year(s) Of Engagement Activity 2018
URL http://gyrokinetics.sourceforge.net/wiki/index.php/GS2_Developers_Meeting_2018
 
Description RSE Conference 2018 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The Third Research Software Engineers (RSE) conference took place in September at Birmingham University. Ania Brown was on the organising committee, responsible for organising the workshops and acting as diversity chair, and I atended as a delegate. Over 300 people attended, and the varied program addressed the wide spectrum of RSEs interests, from the more technical issues of the job to the more political ones, such as raising awareness of RSEs and developing their career paths. It is also an indispensable arena to network with other RSEs and develop new collaborations.
Year(s) Of Engagement Activity 2018
URL https://rse.ac.uk/conf2018/
 
Description RSE Fellowship Workshop - Sindon, Sept 2018 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact EPSRC organised a workshop for the RSE fellows it has funded. Talks were presented on the current status of the fellowships, and input was provided to EPSRC about current issues and future directions for the program
Year(s) Of Engagement Activity 2018
 
Description RSE London South East 2019 workshop, London, UK 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact RSE London/South-east is a regional version of the national RSE conference. Ania Brown used her experience in organising the national conference to help organise this new event. Over 50 people attended, and workshops and talks relevant to RSEs were presented.
Year(s) Of Engagement Activity 2019
URL https://rslondon.ac.uk/rslondonse-2019/
 
Description Researcher's Conference, Department of Engineering Science, University of Oxford 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact I and Ania presented our work to the rest of the Oxford Engineering Department through a talk and a poster. A small number of interactions have grown through thi, partially as due to a local reorganisation we are new members to this department.
Year(s) Of Engagement Activity 2018
 
Description Teach Fortran At Sheffield University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact At the invitation of other RSE fellows I presented a 2 day course on Modern Fortran to researchers so as to enable thme to meet the programming needs of their projects
Year(s) Of Engagement Activity 2017
 
Description UK OpenMP Users Conference, Oxford, UK (21-22/05/2018). 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Ania and I attended the national conference on one of the main techniques we use in our work. We learnt the latest methods, what was coming soon, and got a free book!
Year(s) Of Engagement Activity 2018
URL https://www.openmp.org/blog/uk-openmp-users-conf-2018-report/
 
Description Women in Engineering Symposium, Department of Engineering Science, University of Oxford, UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact Ania presented her work to raise the profile of women work as software engineers
Year(s) Of Engagement Activity 2018