Computing Resources and Software Support for the UKQCD Physics Programme

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Physics and Astronomy

Abstract

QCD is the quantum theory of the strong interaction, formulated in terms of elementary entities called quarks and gluons. In a laboratory, however, we only observe composite objects such as protons (qqq) or pi-mesons ( q -- anti-q), collectively known as hadrons. Explaining this dichotomy in a satisfactory way is a major unresolved question in theoretical physics, and indeed commands a prize of 1 million dollars from the Clay Mathematics Institute. In the interim, many important QCD questions can be addressed using a controlled approximation called lattice gauge theory, in which quarks and gluons are formulated only on the discrete points and links of a spacetime lattice. Properties such as the masses and interactions of hadrons, and the nature of the gluonic flux tube which holds quarks together with a force sufficient to lift 3 elephants can then be calculated using a numerical technique called Monte Carlo Importance Sampling. Precise lattice QCD results are now a crucial ingredient in projects which, by comparing theoretical predictions with the results of collider experiments, aim both to determine parameters of the Standard Model of particle physics, and if inconsistencies are found, to open up the exciting prospect of 'new physics'. Unfortunately, the number of lattice points required to make the calculations with the necessary precision is large, as is the computational effort associated with intermediate steps of the calculation once the quarks are assigned realistically small masses. In this proposal we argue that in order to achieve the required accuracy state-of-the-art high performance computing resources are needed; indeed, we cost our bid in units of 'teraflop-years', equivalent to some 3x10^19 individual computer operations. Lattice QCD is truly a Grand Challenge. The projects presented in this bid cover a broad sweep of different physics, ranging from precision calculations of hadrons; the properties of systems including heavy quarks such as 'charm' and 'bottom'; extracting fundamental parameters of the Standard Model of Particle Physics (such as the masses and decay properties of hadrons); the quark-gluon plasma which forms at temperatures in excess of a trillion kelvin; the exotic 'quark matter' thought to exist in the cores of neutron stars (having the mass of the Sun but compact enough to comfortably fit in Swansea Bay!); finally, the properties of QCD-like models in which the numbers of quarks and gluons differs from their real- world values of 3 and 8, offering the tantalising promise of making contact with pencil-and-paper approaches and ultimately the Clay prize....

Publications

10 25 50
 
Description This was one of a linked set of grants to institutions belonging to the UKQCD Consortium which, together, have been used
to procure distributed high-performance computing (HPC) facilities and associated software to support UKQCD's research
in theoretical particle physics. These facilities were procured and brought into operation successfully, and have been
integrated with others supporting astrophysics to form the DiRAC (Distributed Research utilising Advance Computing)
National Facility.

Edinburgh's primary objective was to develop the BlueGene/Q computer jointly with IBM, and to construct and install a four-rack pre-production system. This was achieved at the end of 2011 at which point BlueGene/Q was the most energy-efficient HPC architecture in the world.

The Edinburgh machine was brought into service early in 2012 for use by UKQCD and has subsequently been upgraded with two additional racks to support wider use by the DiRAC Consortium.

The BlueGene/Q design is already internationally recognised, having taken top place in the Green500 supercomputing list
in November 2010, beating the nearest GPU competitor by a substantial margin, and remaining in top place for two years.
The Livermore Lab BlueGene/Q installation was the fastest computer in the world (top500) in June 2012.
Exploitation Route BlueGene/Q is enabling UKQCD to perform simulations that include quarks with physical masses, using a chirally symmetric fermion action, for the first time. This eliminates the need for a mass extrapolation and gives systematic
improvement of our results for QCD matrix elements that are vital inputs to searches for physics beyond the Standard
Model. We have used the BlueGene/P system procured by Swansea University via this grant to explore models for
electroweak symmetry breaking and this work is continuing on BlueGene/Q. UKQCD is also using BlueGene/Q for
simulations of the quark-gluon plasma phase of QCD.
Sectors Digital/Communication/Information Technologies (including Software)

 
Description In December 2007, Peter Boyle of the University of Edinburgh was invited to lead an international academic team developing the memory prefetch engine for IBM's next-generation BlueGene/Q architecture in a unique academic-industrial collaboration on core IBM technology. This was the subject of a Collaboration Agreement between IBM, the University of Edinburgh and Columbia University. BlueGene/Q has represented roughly a billion dollar project for IBM. This has been a unique experiment in co-design at the cutting edge of technology, with Boyle using his advanced QCD software and silicon design skills to optimise the computer's performance. Many architectural decisions have been influenced by the QCD codes and academic design team members. The prefetch engine is a key performance differentiator and was entirely under Edinburgh's design control. The BlueGene/Q development resulted in the following US Patents pending that were filed by IBM and include Peter Boyle: 20110219208 MULTI-PETASCALE HIGHLY EFFICIENT PARALLEL SUPERCOMPUTER 20110173398 TWO DIFFERENT PREFETCHING COMPLEMENTARY ENGINES OPERATING SIMULTANEOUSLY 20110173397 PROGRAMMABLE STREAM PREFETCH WITH RESOURCE OPTIMIZATION 20110119426 LIST BASED PREFETCH The BlueGene/Q design has been internationally recognised, holding top place in the Green500 supercomputer list for two years from November 2010, and becoming the fastest computer in the world (Top500) in June 2012.
First Year Of Impact 2010
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Cultural,Economic

 
Description PPRP
Amount £6,000,000 (GBP)
Funding ID ST/K000411/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 01/2012 
End 12/2012
 
Description PPRP
Amount £427,192 (GBP)
Funding ID ST/K005804/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 05/2012 
End 04/2015
 
Description Standard (FEC)
Amount £1,693,370 (GBP)
Funding ID ST/J000329/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 10/2011 
End 09/2014
 
Description Standard (FEC)
Amount £1,370,934 (GBP)
Funding ID ST/M006530/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 05/2015 
End 03/2017
 
Description Standard (FEC)
Amount £390,203 (GBP)
Funding ID ST/L000458/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 10/2014 
End 09/2017
 
Description University infrastructure
Amount £156,000 (GBP)
Funding ID DiRAC1 
Organisation University of Edinburgh 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 06/2010 
End 07/2011
 
Title LIST BASED PREFETCH 
Description Invention of a novel gather-prefetch engine for accelerating sparse matrix codes 
IP Reference US8806141 
Protection Patent granted
Year Protection Granted 2014
Licensed Commercial In Confidence
Impact None
 
Title LIST BASED PREFETCH 
Description Invention of a pattern-recording gather prefetch engine 
IP Reference US8255633 
Protection Patent granted
Year Protection Granted 2012
Licensed Commercial In Confidence
Impact Used in IBM's HPC products
 
Title Method and apparatus of parallel computing with simultaneously operating stream prefetching and list prefetching engines 
Description Invention of a prefetch engine capable of executing two distinct algorithms concurrently 
IP Reference US8327077 
Protection Patent granted
Year Protection Granted 2012
Licensed Commercial In Confidence
Impact Widely used feature in IBM's premier HPC systems
 
Title Multi-petascale highly efficient parallel supercomputer 
Description Participation in the invention of the IBM BlueGene/Q compute chip 
IP Reference US9081501 
Protection Patent granted
Year Protection Granted 2015
Licensed Commercial In Confidence
Impact BlueGene has been IBM's premier HPC product for several years
 
Title PROGRAMMABLE STREAM PREFETCH WITH RESOURCE OPTIMIZATION 
Description Invention of an adaptive and intelligent stream prefetch algorithm 
IP Reference US8347039 
Protection Patent granted
Year Protection Granted 2013
Licensed Commercial In Confidence
Impact Exploited successfully in a wide range of scientific codes
 
Description Green500 most energy efficient supercomputer announcement 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Press release announcing that a prototype computer developed jointly with IBM and partially with STFC funding had been ranked the most energy efficient supercomputer in the world.

Various web links to the announcement and reports on the web, but no take up by other media.
Year(s) Of Engagement Activity 2010