Computing Resources and Software Support for the UKQCD Physics Programme

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Physics and Astronomy

Abstract

Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
 
Description This was one of a linked set of grants to institutions belonging to the UKQCD Consortium which, together, have been used
to procure distributed high-performance computing (HPC) facilities and associated software to support UKQCD's research
in theoretical particle physics. These facilities were procured and brought into operation successfully, and have been
integrated with others supporting astrophysics to form the DiRAC (Distributed Research utilising Advance Computing)
National Facility.

Edinburgh's primary objective was to develop the BlueGene/Q computer jointly with IBM, and to construct and install a four-rack pre-production system. This was achieved at the end of 2011 at which point BlueGene/Q was the most energy-efficient HPC architecture in the world.

The Edinburgh machine was brought into service early in 2012 for use by UKQCD and has subsequently been upgraded with two additional racks to support wider use by the DiRAC Consortium.

The BlueGene/Q design is already internationally recognised, having taken top place in the Green500 supercomputing list
in November 2010, beating the nearest GPU competitor by a substantial margin, and remaining in top place for two years.
The Livermore Lab BlueGene/Q installation was the fastest computer in the world (top500) in June 2012.
Exploitation Route BlueGene/Q is enabling UKQCD to perform simulations that include quarks with physical masses, using a chirally symmetric fermion action, for the first time. This eliminates the need for a mass extrapolation and gives systematic
improvement of our results for QCD matrix elements that are vital inputs to searches for physics beyond the Standard
Model. We have used the BlueGene/P system procured by Swansea University via this grant to explore models for
electroweak symmetry breaking and this work is continuing on BlueGene/Q. UKQCD is also using BlueGene/Q for
simulations of the quark-gluon plasma phase of QCD.
Sectors Digital/Communication/Information Technologies (including Software)
 
Description In December 2007, Peter Boyle of the University of Edinburgh was invited to lead an international academic team developing the memory prefetch engine for IBM's next-generation BlueGene/Q architecture in a unique academic-industrial collaboration on core IBM technology. This was the subject of a Collaboration Agreement between IBM, the University of Edinburgh and Columbia University. BlueGene/Q has represented roughly a billion dollar project for IBM. This has been a unique experiment in co-design at the cutting edge of technology, with Boyle using his advanced QCD software and silicon design skills to optimise the computer's performance. Many architectural decisions have been influenced by the QCD codes and academic design team members. The prefetch engine is a key performance differentiator and was entirely under Edinburgh's design control. The BlueGene/Q development resulted in the following US Patents pending that were filed by IBM and include Peter Boyle: 20110219208 MULTI-PETASCALE HIGHLY EFFICIENT PARALLEL SUPERCOMPUTER 20110173398 TWO DIFFERENT PREFETCHING COMPLEMENTARY ENGINES OPERATING SIMULTANEOUSLY 20110173397 PROGRAMMABLE STREAM PREFETCH WITH RESOURCE OPTIMIZATION 20110119426 LIST BASED PREFETCH The BlueGene/Q design has been internationally recognised, holding top place in the Green500 supercomputer list for two years from November 2010, and becoming the fastest computer in the world (Top500) in June 2012.
First Year Of Impact 2010
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Cultural,Economic
 
Description PPRP
Amount £6,000,000 (GBP)
Funding ID ST/K000411/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Public
Country United Kingdom
Start 01/2012 
End 12/2012
 
Description PPRP
Amount £427,192 (GBP)
Funding ID ST/K005804/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Public
Country United Kingdom
Start 05/2012 
End 04/2015
 
Description Standard (FEC)
Amount £2,393,688 (GBP)
Funding ID ST/L000458/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Public
Country United Kingdom
Start 10/2014 
End 09/2017
 
Description Standard (FEC)
Amount £1,370,934 (GBP)
Funding ID ST/M006530/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Public
Country United Kingdom
Start 05/2015 
End 03/2017
 
Description Standard (FEC)
Amount £1,693,370 (GBP)
Funding ID ST/J000329/1 
Organisation Science and Technologies Facilities Council (STFC) 
Sector Public
Country United Kingdom
Start 10/2011 
End 09/2014
 
Description University infrastructure
Amount £156,000 (GBP)
Funding ID DiRAC1 
Organisation University of Edinburgh 
Sector Academic/University
Country United Kingdom
Start 06/2010 
End 07/2011
 
Title LIST BASED PREFETCH 
Description A list prefetch engine improves a performance of a parallel computing system. The list prefetch engine receives a current cache miss address. The list prefetch engine evaluates whether the current cache miss address is valid. If the current cache miss address is valid, the list prefetch engine compares the current cache miss address and a list address. A list address represents an address in a list. A list describes an arbitrary sequence of prior cache miss addresses. The prefetch engine prefetches data according to the list, if there is a match between the current cache miss address and the list address. 
IP Reference US2011119426 
Protection Patent granted
Year Protection Granted 2011
Licensed Commercial In Confidence
Impact Used in IBM's HPC products
 
Title LIST BASED PREFETCH 
Description A list prefetch engine improves a performance of a parallel computing system. The list prefetch engine receives a current cache miss address. The list prefetch engine evaluates whether the current cache miss address is valid. If the current cache miss address is valid, the list prefetch engine compares the current cache miss address and a list address. A list address represents an address in a list. A list describes an arbitrary sequence of prior cache miss addresses. The prefetch engine prefetches data according to the list, if there is a match between the current cache miss address and the list address. 
IP Reference US2012324142 
Protection Patent granted
Year Protection Granted 2012
Licensed Commercial In Confidence
Impact None
 
Title MULTI-PETASCALE HIGHLY EFFICIENT PARALLEL SUPERCOMPUTER 
Description A Multi-Petascale Highly Efficient Parallel Supercomputer of 100 petaOPS-scale computing, at decreased cost, power and footprint, and that allows for a maximum packaging density of processing nodes from an interconnect point of view. The Supercomputer exploits technological advances in VLSI that enables a computing model where many processors can be integrated into a single Application Specific Integrated Circuit (ASIC). Each ASIC computing node comprises a system-on-chip ASIC utilizing four or more processors integrated into one die, with each having full access to all system resources and enabling adaptive partitioning of the processors to functions such as compute or messaging I/O on an application by application basis, and preferably, enable adaptive partitioning of functions in accordance with various algorithmic phases within an application, or if I/O or other processors are underutilized, then can participate in computation or communication nodes are interconnected by a five dimensional torus network with DMA that optimally maximize the throughput of packet communications between nodes and minimize latency. 
IP Reference US2011219208 
Protection Patent granted
Year Protection Granted 2011
Licensed Commercial In Confidence
Impact BlueGene has been IBM's premier HPC product for several years
 
Title PROGRAMMABLE STREAM PREFETCH WITH RESOURCE OPTIMIZATION 
Description A stream prefetch engine performs data retrieval in a parallel computing system. The engine receives a load request from at least one processor. The engine evaluates whether a first memory address requested in the load request is present and valid in a table. The engine checks whether there exists valid data corresponding to the first memory address in an array if the first memory address is present and valid in the table. The engine increments a prefetching depth of a first stream that the first memory address belongs to and fetching a cache line associated with the first memory address from the at least one cache memory device if there is not yet valid data corresponding to the first memory address in the array. The engine determines whether prefetching of additional data is needed for the first stream within its prefetching depth. The engine prefetches the additional data if the prefetching is needed. 
IP Reference US2011173397 
Protection Patent granted
Year Protection Granted 2011
Licensed Commercial In Confidence
Impact Exploited successfully in a wide range of scientific codes
 
Title TWO DIFFERENT PREFETCHING COMPLEMENTARY ENGINES OPERATING SIMULTANEOUSLY 
Description A prefetch system improves a performance of a parallel computing system. The parallel computing system includes a plurality of computing nodes. A computing node includes at least one processor and at least one memory device. The prefetch system includes at least one stream prefetch engine and at least one list prefetch engine. The prefetch system operates those engines simultaneously. After the at least one processor issues a command, the prefetch system passes the command to a stream prefetch engine and a list prefetch engine. The prefetch system operates the stream prefetch engine and the list prefetch engine to prefetch data to be needed in subsequent clock cycles in the processor in response to the passed command. 
IP Reference US2011173398 
Protection Patent granted
Year Protection Granted 2011
Licensed Commercial In Confidence
Impact Widely used feature in IBM's premier HPC systems
 
Description Green500 most energy efficient supercomputer announcement 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Press release announcing that a prototype computer developed jointly with IBM and partially with STFC funding had been ranked the most energy efficient supercomputer in the world.

Various web links to the announcement and reports on the web, but no take up by other media.
Year(s) Of Engagement Activity 2010