DiRAC SMP Facility: Revealing the Structure of the Universe

Lead Research Organisation: University of Cambridge
Department Name: Applied Maths and Theoretical Physics

Abstract

This proposal aims to address key questions about the fundamental structure of the Universe and the origin and nature of the galaxies, stars and planets within it. This proposal is to enhance the STFC DiRAC Facility which provides the primary computational platform for UK particle physicists, cosmologists and astrophysicists. The proposal will fund the DiRAC SMP Node, a highly flexible shared-memory (SMP) system with up to 16TB of globally addressable memory, which will be the largest such system in Europe and the first of its kind in the world to have Many Integrated Core accelerators. This HPC system will aim to advance the study of the early universe, the cosmic microwave sky and extra-solar planets. The unique SMP capabilities of this flexible architecture will be available to the wider community of DiRAC users.

This proposal will advance our understanding in three key scientific areas:

A. Science exploitation of the Cosmic Microwave Sky and Large-scale Structure surveys:

The cosmic microwave background (CMB) remains the premier source for cosmological information. Planck satellite data, in which many COSMOS consortium members have leading roles, dramatically supersedes the previous WMAP satellite. The new SMP node will be actively used to open up new frontiers in CMB science which will constrain fundamental cosmology. With key Planck data releases scheduled for 2013 and 2014, the proposed SMP upgrade would be timed strategically to leverage the proprietary Planck data, maximizing science exploitation by UK members prior to release. Complementary work on large-scale structure will encompass a range of topics, from using surveys to constrain fundamental properties of the Universe, to understanding the hierarchical formation of galaxies and uncovering their detailed properties.

B. Observational consequences of the Early Universe.

The COSMOS consortium has pioneered the use of lattice field theory simulations to understand the physics of non-linear phenomena in the early universe. Such epochs are expected in most cosmological models derived from fundamental theories such as the unification of forces, cosmic defects, extra dimensions, and string and M-theory. The challenge using the new SMP node will be to identify and calculate the observable consequences of these theories, which can range from signatures in the cosmic microwave background or the large-scale structure of the universe through to the production of dark matter or primordial gravitational waves.

C. Extra-solar planets and their atmospheres.

The DiRAC SMP node will also support key UK research in extra-solar planets by the Miracle consortium. This is concerned with the observation and characterisation of exoplanets and their atmospheres, developing key numerical codes which are vital to this international endeavour. The new SMP system will allow much more information to be extracted from spectroscopic data from exoplanet environments. This will help us answer some of the oldest questions in science such as: Are there worlds beyond our solar system? Are they numerous or rare? How many of them have the right conditions for life?

Planned Impact

The DiRAC SMP facility will be a state-of-the-art supercomputer system which at delivery will be:

- Largest shared-memory single-image system in Europe.
- First Intel MIC-enabled SMP system in the world (i.e. Many Integrated Core technology)

From the perspective of UK cosmology and astrophysics research communities, the DiRAC SMP node will be an internationally competitive facility showcasing the next generation of highly flexible user-friendly HPC technologies.

The primary purpose of the DiRAC SMP facility is to enable UK researchers to maintain international
leadership in the investigation of the origin and structure of the Universe. This entails simulation of innovative theoretical ideas and comparison with vast new observational data sets, notably
exploiting maps of the cosmic microwave sky and large galaxy surveys, as well as spectral data from extra-solar planets and their atmospheres.

This work has a significant economic and social impact through (i) training in high performance computing and (ii) collaboration and interaction with HPC technology developers and (iii) public understanding of science and technology. The SMP will have a large user base of 30-50 new users annually (mainly PDRAs and graduate students) and it will maintain a special emphasis on user accessibility and education, ensuring maximum benefit to UK science and subsequent employers. We will also work with commerce, research design labs and other academic areas to produce new IT (hardware and software) solutions, and we will collaborate with leading IT vendors to test and design workload management systems, compilers and performance tools.

DiRAC SMP node users have a high public profile and will continue an active programme of outreach through open days, print media, internet, radio and television. Cosmology and exoplanets are areas of major public interest.

Publications

10 25 50
 
Description High energy grazing collisions of black holes
Amount $48,207 (USD)
Funding ID NSF-XSEDE Grant No. PHY-090003 
Organisation National Science Foundation (NSF) 
Sector Public
Country United States
Start 01/2017 
End 12/2017
 
Description NSF XSEDE Grant PHY-090003
Amount $59,225 (USD)
Organisation National Science Foundation (NSF) 
Sector Public
Country United States
Start 01/2014 
End 12/2014
 
Description Probing fundamental physics with gravity
Amount € 1 (EUR)
Funding ID RACE grant Tier-0 PPFPWG 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 12/2016 
End 11/2017
 
Description Intel/SGI/COSMOS Collaboration 
Organisation Intel Corporation
Country United States 
Sector Private 
PI Contribution The present relationship is a longstanding collaboration and partnership in high performance computing with SGI since 1997 and with Intel since 2003. On the basis of taking delivery of the largest deployment of Xeon Phi processors in a single-image shared memory system, we have been testing the architecture and system stack while developing codes to exploit this new many-integrated core (MIC) technology. We influenced design of the initial "MG" blade accommodating the Xeon Phi within the SGI UV2, and we have worked with Intel to improve their Xeon Phi offload implementation.
Collaborator Contribution Our partner SGI has contributed half the salary GPB£18000 so that COSMOS could employ a parallel programmer, James Briggs. Intel have contributed 50% of an Intel employee, John Pennycook, to assist with development work and to act as a direct link to Intel software engineering teams. SGI has also contributed substantial software consultancy support through Cheng Liao which has led to new scalable SMP applications.
Impact There have been many positive outcomes from this collaboration, notably the Xeon Phi acceleration of important COSMOS consortium codes, including Camb (main CMB cosmological parameter estimation code), WALLS (early universe field theory code), MODAL (non-Gaussianity code), etc. A major development has been the implementation of a scalable shared-memory eigensolver capable of decomposing dense matrices up to 1M x 1M; this has important applications for the TROVE pipeline, determining exoplanet spectral lines (and elsewhere). Work from this collaboration was reported at Supercomputing 2013, held in Denver, November 2013.
Start Year 2013
 
Description Intel/SGI/COSMOS Collaboration 
Organisation Silicon Graphics Inc
Country United States 
Sector Private 
PI Contribution The present relationship is a longstanding collaboration and partnership in high performance computing with SGI since 1997 and with Intel since 2003. On the basis of taking delivery of the largest deployment of Xeon Phi processors in a single-image shared memory system, we have been testing the architecture and system stack while developing codes to exploit this new many-integrated core (MIC) technology. We influenced design of the initial "MG" blade accommodating the Xeon Phi within the SGI UV2, and we have worked with Intel to improve their Xeon Phi offload implementation.
Collaborator Contribution Our partner SGI has contributed half the salary GPB£18000 so that COSMOS could employ a parallel programmer, James Briggs. Intel have contributed 50% of an Intel employee, John Pennycook, to assist with development work and to act as a direct link to Intel software engineering teams. SGI has also contributed substantial software consultancy support through Cheng Liao which has led to new scalable SMP applications.
Impact There have been many positive outcomes from this collaboration, notably the Xeon Phi acceleration of important COSMOS consortium codes, including Camb (main CMB cosmological parameter estimation code), WALLS (early universe field theory code), MODAL (non-Gaussianity code), etc. A major development has been the implementation of a scalable shared-memory eigensolver capable of decomposing dense matrices up to 1M x 1M; this has important applications for the TROVE pipeline, determining exoplanet spectral lines (and elsewhere). Work from this collaboration was reported at Supercomputing 2013, held in Denver, November 2013.
Start Year 2013
 
Description 10th Rencontres du Vietnam 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 10th Rencontres du Vietnam Very High Energy Phenomena in the Universe
Quy Nhon, Vietnam, 8 August 2014

Talk title: "Numerical simulations of coalescing binaries"
Year(s) Of Engagement Activity 2014
 
Description 3rd Sant Cugat Forum on Astrophysics 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 3rd Sant Cugat Forum on Astrophysics
Sant Cugat, Spain, 25 April 2014

Talk title: "Gravitational Recoil and Astrophysical impact"
Year(s) Of Engagement Activity 2014
 
Description 6th form talk (Chelmsford) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact October 2017: Ulrich Sperhake gave a talk to a 6th form college in Chelmsford. Subject: The direct detection of gravitational waves by LIGO and the insights already obtained from this new window on the Universe as well as the enormous potential for our search for answers to the most profound questions in astrophysics and fundamental physics.
Year(s) Of Engagement Activity 2017
 
Description Comenius 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact John D. Barrow gave four talks on cosmology to Comenius (Netherlands) University group, March and September 2014, with two more to come in November 2014

See above
Year(s) Of Engagement Activity 2014
 
Description Foligno 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact John D. Barrow gave two talks at Foligno Science Festival, Italy, April 2014

See above
Year(s) Of Engagement Activity 2014
 
Description IMO (Cape Town) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact IMO Celebrity Lecture, The Maths of Entire Universes, International Mathematics Olympiad, U Cape Town, 2014

The International Mathematical Olympiad (IMO) is the World Championship Mathematics Competition for High School students and is held annually in a different country.
Year(s) Of Engagement Activity 2014
 
Description IOP Engineering Society (Chelmsford) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact October 2017: Ulrich Sperhake gave a talk to the IOP Engineering Society in Chelmsford. Subject: The direct detection of gravitational waves by LIGO and the insights already obtained from this new window on the Universe as well as the enormous potential for our search for answers to the most profound questions in astrophysics and fundamental physics.
Year(s) Of Engagement Activity 2017
 
Description Lecture (Isaac Newton Institute) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Annual Correspondents Day Lecture, Isaac Newton Institute, Cambridge, 2014

The lecture was attended by representatives from EPSRC, BBSRC, ESRC, NERC and STFC. The lecture is important in maintaining and increasing the value of the Institute to the entire UK mathematical community
Year(s) Of Engagement Activity 2014
 
Description New Perspectives on Cosmology 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Yi Wang gave a talk entitled "CMB Anomalies in the Light of BICEP2" at the IAS Workshop, "New Perspectives on Cosmology," 23 May 2014 at the Hong Kong University of Science and Technology

During last few years we have witnessed rapid developments in cosmology, both in theory and experiments. Especially, with the recent PLANCK data our understanding on the universe is facing even more refinements than ever. Moreover, with the planned number of future experiments on the cosmic microwave background, large scale structure, gravitational waves and so on, our view on cosmology and gravity will be further developed continuously. The aim of the workshop was to promote stimulating informal and intensive discussions among the participants and to initiate possible collaborations.
Year(s) Of Engagement Activity 2014
URL http://ias.ust.hk/workshop/cosmology/2014/index.html
 
Description Pordenone 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact John D. Barrow gave a talk and a formal interview at Pordenone Literary Festival, Italy, September 2014

See above
Year(s) Of Engagement Activity 2014
URL http://www.oppureblog.com/john-barrow-ci-racconta-cose-linfinito/
 
Description Rollins 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact John D. Barrow gave a talk at Rollins College Florida, "Origin and Evolution of the Universe," October 2014

See above
Year(s) Of Engagement Activity 2014
 
Description Science Festival 2013 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Substantial effort was made during the Cambridge Science Festival to publicise the role of the COSMOS supercomputer at the Centre for Mathematical Sciences, where hundreds of members of the general public visited on Saturday 23 March 2013. Group tours were offered
every half hour which proved a particular highlight for the Open Day. The role of COSMOS for key science exploitation of
the Planck satellite (announced on 21st March with considerable media coverage) was also emphasised.

Considerable interest shown by members of the public.
Year(s) Of Engagement Activity 2013
 
Description Vice Chancellor, Cape Town 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact John D. Barrow gave a talk in Vice Chancellor's series, University of Cape Town, "Origin and Evolution of the Universe," July 2014

See above
Year(s) Of Engagement Activity 2014