Computing Resources and Software Support for the UKQCD Physics Programme
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Physics and Astronomy
Abstract
QCD is the quantum theory of the strong interaction, formulated in terms of elementary entities called quarks and gluons. In a laboratory, however, we only observe composite objects such as protons (qqq) or pi-mesons ( q -- anti-q), collectively known as hadrons. Explaining this dichotomy in a satisfactory way is a major unresolved question in theoretical physics, and indeed commands a prize of 1 million dollars from the Clay Mathematics Institute. In the interim, many important QCD questions can be addressed using a controlled approximation called lattice gauge theory, in which quarks and gluons are formulated only on the discrete points and links of a spacetime lattice. Properties such as the masses and interactions of hadrons, and the nature of the gluonic flux tube which holds quarks together with a force sufficient to lift 3 elephants can then be calculated using a numerical technique called Monte Carlo Importance Sampling. Precise lattice QCD results are now a crucial ingredient in projects which, by comparing theoretical predictions with the results of collider experiments, aim both to determine parameters of the Standard Model of particle physics, and if inconsistencies are found, to open up the exciting prospect of 'new physics'. Unfortunately, the number of lattice points required to make the calculations with the necessary precision is large, as is the computational effort associated with intermediate steps of the calculation once the quarks are assigned realistically small masses. In this proposal we argue that in order to achieve the required accuracy state-of-the-art high performance computing resources are needed; indeed, we cost our bid in units of 'teraflop-years', equivalent to some 3x10^19 individual computer operations. Lattice QCD is truly a Grand Challenge. The projects presented in this bid cover a broad sweep of different physics, ranging from precision calculations of hadrons; the properties of systems including heavy quarks such as 'charm' and 'bottom'; extracting fundamental parameters of the Standard Model of Particle Physics (such as the masses and decay properties of hadrons); the quark-gluon plasma which forms at temperatures in excess of a trillion kelvin; the exotic 'quark matter' thought to exist in the cores of neutron stars (having the mass of the Sun but compact enough to comfortably fit in Swansea Bay!); finally, the properties of QCD-like models in which the numbers of quarks and gluons differs from their real- world values of 3 and 8, offering the tantalising promise of making contact with pencil-and-paper approaches and ultimately the Clay prize....
Publications

Aarts G
(2013)
S wave bottomonium states moving in a quark-gluon plasma from lattice NRQCD
in Journal of High Energy Physics

Aarts G
(2011)
What happens to the $ \Upsilon $ and ? b in the quark-gluon plasma? Bottomonium spectral functions from lattice QCD
in Journal of High Energy Physics

Aarts G
(2013)
Melting of P wave bottomonium states in the quark-gluon plasma from lattice NRQCD
in Journal of High Energy Physics

Aarts G
(2015)
Nucleons and parity doubling across the deconfinement transition
in Physical Review D

Aarts G
(2017)
Parity doubling of nucléons, Delta and Omega baryons across the deconfinement phase transition
in EPJ Web of Conferences

Aarts G
(2016)
Complex Langevin in Lattice QCD: Dynamic Stabilisation and the Phase Diagram
in Acta Physica Polonica B Proceedings Supplement

Aarts G
(2015)
Electrical conductivity and charge diffusion in thermal QCD from the lattice
in Journal of High Energy Physics

Aarts G
(2016)
Finite Temperature Lattice QCD --- Baryons in the Quark--Gluon Plasma
in Acta Physica Polonica B Proceedings Supplement

Aarts G
(2014)
The bottomonium spectrum at finite temperature from N f = 2 + 1 lattice QCD
in Journal of High Energy Physics
Description | This was one of a linked set of grants to institutions belonging to the UKQCD Consortium which, together, have been used to procure distributed high-performance computing (HPC) facilities and associated software to support UKQCD's research in theoretical particle physics. These facilities were procured and brought into operation successfully, and have been integrated with others supporting astrophysics to form the DiRAC (Distributed Research utilising Advance Computing) National Facility. Edinburgh's primary objective was to develop the BlueGene/Q computer jointly with IBM, and to construct and install a four-rack pre-production system. This was achieved at the end of 2011 at which point BlueGene/Q was the most energy-efficient HPC architecture in the world. The Edinburgh machine was brought into service early in 2012 for use by UKQCD and has subsequently been upgraded with two additional racks to support wider use by the DiRAC Consortium. The BlueGene/Q design is already internationally recognised, having taken top place in the Green500 supercomputing list in November 2010, beating the nearest GPU competitor by a substantial margin, and remaining in top place for two years. The Livermore Lab BlueGene/Q installation was the fastest computer in the world (top500) in June 2012. |
Exploitation Route | BlueGene/Q is enabling UKQCD to perform simulations that include quarks with physical masses, using a chirally symmetric fermion action, for the first time. This eliminates the need for a mass extrapolation and gives systematic improvement of our results for QCD matrix elements that are vital inputs to searches for physics beyond the Standard Model. We have used the BlueGene/P system procured by Swansea University via this grant to explore models for electroweak symmetry breaking and this work is continuing on BlueGene/Q. UKQCD is also using BlueGene/Q for simulations of the quark-gluon plasma phase of QCD. |
Sectors | Digital/Communication/Information Technologies (including Software) |
Description | In December 2007, Peter Boyle of the University of Edinburgh was invited to lead an international academic team developing the memory prefetch engine for IBM's next-generation BlueGene/Q architecture in a unique academic-industrial collaboration on core IBM technology. This was the subject of a Collaboration Agreement between IBM, the University of Edinburgh and Columbia University. BlueGene/Q has represented roughly a billion dollar project for IBM. This has been a unique experiment in co-design at the cutting edge of technology, with Boyle using his advanced QCD software and silicon design skills to optimise the computer's performance. Many architectural decisions have been influenced by the QCD codes and academic design team members. The prefetch engine is a key performance differentiator and was entirely under Edinburgh's design control. The BlueGene/Q development resulted in the following US Patents pending that were filed by IBM and include Peter Boyle: 20110219208 MULTI-PETASCALE HIGHLY EFFICIENT PARALLEL SUPERCOMPUTER 20110173398 TWO DIFFERENT PREFETCHING COMPLEMENTARY ENGINES OPERATING SIMULTANEOUSLY 20110173397 PROGRAMMABLE STREAM PREFETCH WITH RESOURCE OPTIMIZATION 20110119426 LIST BASED PREFETCH The BlueGene/Q design has been internationally recognised, holding top place in the Green500 supercomputer list for two years from November 2010, and becoming the fastest computer in the world (Top500) in June 2012. |
First Year Of Impact | 2010 |
Sector | Digital/Communication/Information Technologies (including Software) |
Impact Types | Cultural,Economic |
Description | PPRP |
Amount | £6,000,000 (GBP) |
Funding ID | ST/K000411/1 |
Organisation | Science and Technologies Facilities Council (STFC) |
Sector | Public |
Country | United Kingdom |
Start | 01/2012 |
End | 12/2012 |
Description | PPRP |
Amount | £427,192 (GBP) |
Funding ID | ST/K005804/1 |
Organisation | Science and Technologies Facilities Council (STFC) |
Sector | Public |
Country | United Kingdom |
Start | 04/2012 |
End | 04/2015 |
Description | Standard (FEC) |
Amount | £2,393,688 (GBP) |
Funding ID | ST/L000458/1 |
Organisation | Science and Technologies Facilities Council (STFC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2014 |
End | 09/2017 |
Description | Standard (FEC) |
Amount | £1,693,370 (GBP) |
Funding ID | ST/J000329/1 |
Organisation | Science and Technologies Facilities Council (STFC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2011 |
End | 09/2014 |
Description | Standard (FEC) |
Amount | £1,370,934 (GBP) |
Funding ID | ST/M006530/1 |
Organisation | Science and Technologies Facilities Council (STFC) |
Sector | Public |
Country | United Kingdom |
Start | 04/2015 |
End | 03/2017 |
Description | University infrastructure |
Amount | £156,000 (GBP) |
Funding ID | DiRAC1 |
Organisation | University of Edinburgh |
Sector | Academic/University |
Country | United Kingdom |
Start | 05/2010 |
End | 07/2011 |
Title | LIST BASED PREFETCH |
Description | A list prefetch engine improves a performance of a parallel computing system. The list prefetch engine receives a current cache miss address. The list prefetch engine evaluates whether the current cache miss address is valid. If the current cache miss address is valid, the list prefetch engine compares the current cache miss address and a list address. A list address represents an address in a list. A list describes an arbitrary sequence of prior cache miss addresses. The prefetch engine prefetches data according to the list, if there is a match between the current cache miss address and the list address. |
IP Reference | US2011119426 |
Protection | Patent granted |
Year Protection Granted | 2011 |
Licensed | Commercial In Confidence |
Impact | Used in IBM's HPC products |
Title | LIST BASED PREFETCH |
Description | A list prefetch engine improves a performance of a parallel computing system. The list prefetch engine receives a current cache miss address. The list prefetch engine evaluates whether the current cache miss address is valid. If the current cache miss address is valid, the list prefetch engine compares the current cache miss address and a list address. A list address represents an address in a list. A list describes an arbitrary sequence of prior cache miss addresses. The prefetch engine prefetches data according to the list, if there is a match between the current cache miss address and the list address. |
IP Reference | US2012324142 |
Protection | Patent granted |
Year Protection Granted | 2012 |
Licensed | Commercial In Confidence |
Impact | None |
Title | MULTI-PETASCALE HIGHLY EFFICIENT PARALLEL SUPERCOMPUTER |
Description | A Multi-Petascale Highly Efficient Parallel Supercomputer of 100 petaOPS-scale computing, at decreased cost, power and footprint, and that allows for a maximum packaging density of processing nodes from an interconnect point of view. The Supercomputer exploits technological advances in VLSI that enables a computing model where many processors can be integrated into a single Application Specific Integrated Circuit (ASIC). Each ASIC computing node comprises a system-on-chip ASIC utilizing four or more processors integrated into one die, with each having full access to all system resources and enabling adaptive partitioning of the processors to functions such as compute or messaging I/O on an application by application basis, and preferably, enable adaptive partitioning of functions in accordance with various algorithmic phases within an application, or if I/O or other processors are underutilized, then can participate in computation or communication nodes are interconnected by a five dimensional torus network with DMA that optimally maximize the throughput of packet communications between nodes and minimize latency. |
IP Reference | US2011219208 |
Protection | Patent granted |
Year Protection Granted | 2011 |
Licensed | Commercial In Confidence |
Impact | BlueGene has been IBM's premier HPC product for several years |
Title | PROGRAMMABLE STREAM PREFETCH WITH RESOURCE OPTIMIZATION |
Description | A stream prefetch engine performs data retrieval in a parallel computing system. The engine receives a load request from at least one processor. The engine evaluates whether a first memory address requested in the load request is present and valid in a table. The engine checks whether there exists valid data corresponding to the first memory address in an array if the first memory address is present and valid in the table. The engine increments a prefetching depth of a first stream that the first memory address belongs to and fetching a cache line associated with the first memory address from the at least one cache memory device if there is not yet valid data corresponding to the first memory address in the array. The engine determines whether prefetching of additional data is needed for the first stream within its prefetching depth. The engine prefetches the additional data if the prefetching is needed. |
IP Reference | US2011173397 |
Protection | Patent granted |
Year Protection Granted | 2011 |
Licensed | Commercial In Confidence |
Impact | Exploited successfully in a wide range of scientific codes |
Title | TWO DIFFERENT PREFETCHING COMPLEMENTARY ENGINES OPERATING SIMULTANEOUSLY |
Description | A prefetch system improves a performance of a parallel computing system. The parallel computing system includes a plurality of computing nodes. A computing node includes at least one processor and at least one memory device. The prefetch system includes at least one stream prefetch engine and at least one list prefetch engine. The prefetch system operates those engines simultaneously. After the at least one processor issues a command, the prefetch system passes the command to a stream prefetch engine and a list prefetch engine. The prefetch system operates the stream prefetch engine and the list prefetch engine to prefetch data to be needed in subsequent clock cycles in the processor in response to the passed command. |
IP Reference | US2011173398 |
Protection | Patent granted |
Year Protection Granted | 2011 |
Licensed | Commercial In Confidence |
Impact | Widely used feature in IBM's premier HPC systems |
Description | Green500 most energy efficient supercomputer announcement |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Press release announcing that a prototype computer developed jointly with IBM and partially with STFC funding had been ranked the most energy efficient supercomputer in the world. Various web links to the announcement and reports on the web, but no take up by other media. |
Year(s) Of Engagement Activity | 2010 |