DiRAC SMP Facility: Revealing the Structure of the Universe
Lead Research Organisation:
University of Cambridge
Department Name: Applied Maths and Theoretical Physics
Abstract
This proposal aims to address key questions about the fundamental structure of the Universe and the origin and nature of the galaxies, stars and planets within it. This proposal is to enhance the STFC DiRAC Facility which provides the primary computational platform for UK particle physicists, cosmologists and astrophysicists. The proposal will fund the DiRAC SMP Node, a highly flexible shared-memory (SMP) system with up to 16TB of globally addressable memory, which will be the largest such system in Europe and the first of its kind in the world to have Many Integrated Core accelerators. This HPC system will aim to advance the study of the early universe, the cosmic microwave sky and extra-solar planets. The unique SMP capabilities of this flexible architecture will be available to the wider community of DiRAC users.
This proposal will advance our understanding in three key scientific areas:
A. Science exploitation of the Cosmic Microwave Sky and Large-scale Structure surveys:
The cosmic microwave background (CMB) remains the premier source for cosmological information. Planck satellite data, in which many COSMOS consortium members have leading roles, dramatically supersedes the previous WMAP satellite. The new SMP node will be actively used to open up new frontiers in CMB science which will constrain fundamental cosmology. With key Planck data releases scheduled for 2013 and 2014, the proposed SMP upgrade would be timed strategically to leverage the proprietary Planck data, maximizing science exploitation by UK members prior to release. Complementary work on large-scale structure will encompass a range of topics, from using surveys to constrain fundamental properties of the Universe, to understanding the hierarchical formation of galaxies and uncovering their detailed properties.
B. Observational consequences of the Early Universe.
The COSMOS consortium has pioneered the use of lattice field theory simulations to understand the physics of non-linear phenomena in the early universe. Such epochs are expected in most cosmological models derived from fundamental theories such as the unification of forces, cosmic defects, extra dimensions, and string and M-theory. The challenge using the new SMP node will be to identify and calculate the observable consequences of these theories, which can range from signatures in the cosmic microwave background or the large-scale structure of the universe through to the production of dark matter or primordial gravitational waves.
C. Extra-solar planets and their atmospheres.
The DiRAC SMP node will also support key UK research in extra-solar planets by the Miracle consortium. This is concerned with the observation and characterisation of exoplanets and their atmospheres, developing key numerical codes which are vital to this international endeavour. The new SMP system will allow much more information to be extracted from spectroscopic data from exoplanet environments. This will help us answer some of the oldest questions in science such as: Are there worlds beyond our solar system? Are they numerous or rare? How many of them have the right conditions for life?
This proposal will advance our understanding in three key scientific areas:
A. Science exploitation of the Cosmic Microwave Sky and Large-scale Structure surveys:
The cosmic microwave background (CMB) remains the premier source for cosmological information. Planck satellite data, in which many COSMOS consortium members have leading roles, dramatically supersedes the previous WMAP satellite. The new SMP node will be actively used to open up new frontiers in CMB science which will constrain fundamental cosmology. With key Planck data releases scheduled for 2013 and 2014, the proposed SMP upgrade would be timed strategically to leverage the proprietary Planck data, maximizing science exploitation by UK members prior to release. Complementary work on large-scale structure will encompass a range of topics, from using surveys to constrain fundamental properties of the Universe, to understanding the hierarchical formation of galaxies and uncovering their detailed properties.
B. Observational consequences of the Early Universe.
The COSMOS consortium has pioneered the use of lattice field theory simulations to understand the physics of non-linear phenomena in the early universe. Such epochs are expected in most cosmological models derived from fundamental theories such as the unification of forces, cosmic defects, extra dimensions, and string and M-theory. The challenge using the new SMP node will be to identify and calculate the observable consequences of these theories, which can range from signatures in the cosmic microwave background or the large-scale structure of the universe through to the production of dark matter or primordial gravitational waves.
C. Extra-solar planets and their atmospheres.
The DiRAC SMP node will also support key UK research in extra-solar planets by the Miracle consortium. This is concerned with the observation and characterisation of exoplanets and their atmospheres, developing key numerical codes which are vital to this international endeavour. The new SMP system will allow much more information to be extracted from spectroscopic data from exoplanet environments. This will help us answer some of the oldest questions in science such as: Are there worlds beyond our solar system? Are they numerous or rare? How many of them have the right conditions for life?
Planned Impact
The DiRAC SMP facility will be a state-of-the-art supercomputer system which at delivery will be:
- Largest shared-memory single-image system in Europe.
- First Intel MIC-enabled SMP system in the world (i.e. Many Integrated Core technology)
From the perspective of UK cosmology and astrophysics research communities, the DiRAC SMP node will be an internationally competitive facility showcasing the next generation of highly flexible user-friendly HPC technologies.
The primary purpose of the DiRAC SMP facility is to enable UK researchers to maintain international
leadership in the investigation of the origin and structure of the Universe. This entails simulation of innovative theoretical ideas and comparison with vast new observational data sets, notably
exploiting maps of the cosmic microwave sky and large galaxy surveys, as well as spectral data from extra-solar planets and their atmospheres.
This work has a significant economic and social impact through (i) training in high performance computing and (ii) collaboration and interaction with HPC technology developers and (iii) public understanding of science and technology. The SMP will have a large user base of 30-50 new users annually (mainly PDRAs and graduate students) and it will maintain a special emphasis on user accessibility and education, ensuring maximum benefit to UK science and subsequent employers. We will also work with commerce, research design labs and other academic areas to produce new IT (hardware and software) solutions, and we will collaborate with leading IT vendors to test and design workload management systems, compilers and performance tools.
DiRAC SMP node users have a high public profile and will continue an active programme of outreach through open days, print media, internet, radio and television. Cosmology and exoplanets are areas of major public interest.
- Largest shared-memory single-image system in Europe.
- First Intel MIC-enabled SMP system in the world (i.e. Many Integrated Core technology)
From the perspective of UK cosmology and astrophysics research communities, the DiRAC SMP node will be an internationally competitive facility showcasing the next generation of highly flexible user-friendly HPC technologies.
The primary purpose of the DiRAC SMP facility is to enable UK researchers to maintain international
leadership in the investigation of the origin and structure of the Universe. This entails simulation of innovative theoretical ideas and comparison with vast new observational data sets, notably
exploiting maps of the cosmic microwave sky and large galaxy surveys, as well as spectral data from extra-solar planets and their atmospheres.
This work has a significant economic and social impact through (i) training in high performance computing and (ii) collaboration and interaction with HPC technology developers and (iii) public understanding of science and technology. The SMP will have a large user base of 30-50 new users annually (mainly PDRAs and graduate students) and it will maintain a special emphasis on user accessibility and education, ensuring maximum benefit to UK science and subsequent employers. We will also work with commerce, research design labs and other academic areas to produce new IT (hardware and software) solutions, and we will collaborate with leading IT vendors to test and design workload management systems, compilers and performance tools.
DiRAC SMP node users have a high public profile and will continue an active programme of outreach through open days, print media, internet, radio and television. Cosmology and exoplanets are areas of major public interest.
People |
ORCID iD |
Edward Shellard (Principal Investigator) |
Publications
Zilhão M
(2014)
Collisions of oppositely charged black holes
in Physical Review D
Zilhão M
(2014)
Relativity and Gravitation - 100 Years after Einstein in Prague
Zilhão M
(2014)
Testing the nonlinear stability of Kerr-Newman black holes
in Physical Review D
Zilhão M
(2015)
Nonlinear interactions between black holes and Proca fields
in Classical and Quantum Gravity
Zicher N
(2022)
One year of AU Mic with HARPS - I. Measuring the masses of the two transiting planets
in Monthly Notices of the Royal Astronomical Society
Ziampras A
(2023)
Hydrodynamic turbulence in disks with embedded planets
in Astronomy & Astrophysics
Zhu Y
(2022)
Long Dark Gaps in the Lyß Forest at z < 6: Evidence of Ultra-late Reionization from XQR-30 Spectra
in The Astrophysical Journal
Zheng Y
(2022)
Rapidly quenched galaxies in the Simba cosmological simulation and observations
in Monthly Notices of the Royal Astronomical Society
Zhao G
(2013)
The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: weighing the neutrino mass using the galaxy power spectrum of the CMASS sample
in Monthly Notices of the Royal Astronomical Society
Zhang L
(2022)
VIB5 database with accurate ab initio quantum chemical molecular potential energy surfaces.
in Scientific data
Zhang H
(2022)
Spherical accretion of collisional gas in modified gravity I: self-similar solutions and a new cosmological hydrodynamical code
in Monthly Notices of the Royal Astronomical Society
Zenocratti L
(2022)
The origin of correlations between mass, metallicity, and morphology in galaxies from the eagle simulation
in Monthly Notices of the Royal Astronomical Society
Zarrouk P
(2022)
Preliminary clustering properties of the DESI BGS bright targets using DR9 Legacy Imaging Surveys
in Monthly Notices of the Royal Astronomical Society
Yurchenko SN
(2017)
A hybrid line list for CH4 and hot methane continuum.
in Astronomy and astrophysics
Yurchenko SN
(2020)
Treating linear molecules in calculations of rotation-vibration spectra.
in The Journal of chemical physics
Yurchenko SN
(2017)
Symmetry-Adapted Ro-vibrational Basis Functions for Variational Nuclear Motion Calculations: TROVE Approach.
in Journal of chemical theory and computation
Yurchenko SN
(2014)
Spectrum of hot methane in astronomical objects using a comprehensive computed line list.
in Proceedings of the National Academy of Sciences of the United States of America
Yurchenko S
(2013)
Vibrational transition moments of CH4 from first principles
in Journal of Molecular Spectroscopy
Yurchenko S
(2020)
ExoMol line lists - XL. Rovibrational molecular line list for the hydronium ion (H3O+)
in Monthly Notices of the Royal Astronomical Society
Yurchenko S
(2020)
ExoMol line lists - XXXIX. Ro-vibrational molecular line list for CO2
in Monthly Notices of the Royal Astronomical Society
Yurchenko S
(2014)
ExoMol line lists - IV. The rotation-vibration spectrum of methane up to 1500 K
in Monthly Notices of the Royal Astronomical Society
Yurchenko S
(2018)
ExoMol line lists XXXI: spectroscopy of lowest eights electronic states of C2
in Monthly Notices of the Royal Astronomical Society
Yurchenko S
(2020)
ExoMol line lists - XXXVIII. High-temperature molecular line list of silicon dioxide (SiO2)
in Monthly Notices of the Royal Astronomical Society
Young M
(2016)
Quantification of stochastic fragmentation of self-gravitating discs
in Monthly Notices of the Royal Astronomical Society
Young M
(2015)
Binary accretion rates: dependence on temperature and mass ratio
in Monthly Notices of the Royal Astronomical Society
Young A
(2022)
Characteristics of small protoplanetary disc warps in kinematic observations
in Monthly Notices of the Royal Astronomical Society
Yoo T
(2020)
On the origin of low escape fractions of ionizing radiation from massive star-forming galaxies at high redshift
in Monthly Notices of the Royal Astronomical Society
Yang T
(2022)
Understanding the relation between thermal Sunyaev-Zeldovich decrement and halo mass using the simba and TNG simulations
in Monthly Notices of the Royal Astronomical Society
Yachmenev A
(2022)
The nuclear-spin-forbidden rovibrational transitions of water from first principles.
in The Journal of chemical physics
Xu W
(2020)
Galaxy properties in the cosmic web of EAGLE simulation
in Monthly Notices of the Royal Astronomical Society
Wyper P
(2022)
The Imprint of Intermittent Interchange Reconnection on the Solar Wind
in The Astrophysical Journal Letters
Wurster J
(2020)
Non-ideal magnetohydrodynamics versus turbulence - I. Which is the dominant process in protostellar disc formation?
in Monthly Notices of the Royal Astronomical Society
Wurster J
(2022)
On the origin of magnetic fields in stars - II. The effect of numerical resolution
in Monthly Notices of the Royal Astronomical Society
Wurster J
(2021)
Do we need non-ideal magnetohydrodynamic to model protostellar discs?
in Monthly Notices of the Royal Astronomical Society
Wurster J
(2020)
Non-ideal magnetohydrodynamics versus turbulence II: Which is the dominant process in stellar core formation?
in Monthly Notices of the Royal Astronomical Society
Wu X
(2020)
Photometric properties of reionization-epoch galaxies in the simba simulations
in Monthly Notices of the Royal Astronomical Society
Wright S
(2022)
Non-local thermal equilibrium spectra of atmospheric molecules for exoplanets
in Monthly Notices of the Royal Astronomical Society
Woss A
(2020)
Efficient solution of the multichannel Lüscher determinant condition through eigenvalue decomposition
in Physical Review D
Wong A
(2017)
ExoMol line list - XXI. Nitric Oxide (NO)
in Monthly Notices of the Royal Astronomical Society
Witek H
(2020)
Towards numerical relativity in scalar Gauss-Bonnet gravity: 3 + 1 decomposition beyond the small-coupling limit
in Physical Review D
Witek H
(2013)
Numerical relativity in higher-dimensional space-times
Witek H
(2014)
Higher dimensional Numerical Relativity: code comparison
Witek H
(2013)
Superradiant instabilities in astrophysical systems
in Physical Review D
Witek H
(2014)
Higher dimensional numerical relativity: Code comparison
in Physical Review D
WITEK H
(2013)
NUMERICAL RELATIVITY IN HIGHER-DIMENSIONAL SPACE-TIMES
in International Journal of Modern Physics A
Wilson B
(2022)
A measurement of the Ly ß forest power spectrum and its cross with the Ly a forest in X-Shooter XQ-100
in Monthly Notices of the Royal Astronomical Society
Description | High energy grazing collisions of black holes |
Amount | $48,207 (USD) |
Funding ID | NSF-XSEDE Grant No. PHY-090003 |
Organisation | National Science Foundation (NSF) |
Sector | Public |
Country | United States |
Start | 01/2017 |
End | 12/2017 |
Description | NSF XSEDE Grant PHY-090003 |
Amount | $59,225 (USD) |
Organisation | National Science Foundation (NSF) |
Sector | Public |
Country | United States |
Start | 01/2014 |
End | 12/2014 |
Description | Probing fundamental physics with gravity |
Amount | € 1 (EUR) |
Funding ID | RACE grant Tier-0 PPFPWG |
Organisation | European Commission |
Sector | Public |
Country | European Union (EU) |
Start | 12/2016 |
End | 11/2017 |
Description | Intel/SGI/COSMOS Collaboration |
Organisation | Intel Corporation |
Country | United States |
Sector | Private |
PI Contribution | The present relationship is a longstanding collaboration and partnership in high performance computing with SGI since 1997 and with Intel since 2003. On the basis of taking delivery of the largest deployment of Xeon Phi processors in a single-image shared memory system, we have been testing the architecture and system stack while developing codes to exploit this new many-integrated core (MIC) technology. We influenced design of the initial "MG" blade accommodating the Xeon Phi within the SGI UV2, and we have worked with Intel to improve their Xeon Phi offload implementation. |
Collaborator Contribution | Our partner SGI has contributed half the salary GPB£18000 so that COSMOS could employ a parallel programmer, James Briggs. Intel have contributed 50% of an Intel employee, John Pennycook, to assist with development work and to act as a direct link to Intel software engineering teams. SGI has also contributed substantial software consultancy support through Cheng Liao which has led to new scalable SMP applications. |
Impact | There have been many positive outcomes from this collaboration, notably the Xeon Phi acceleration of important COSMOS consortium codes, including Camb (main CMB cosmological parameter estimation code), WALLS (early universe field theory code), MODAL (non-Gaussianity code), etc. A major development has been the implementation of a scalable shared-memory eigensolver capable of decomposing dense matrices up to 1M x 1M; this has important applications for the TROVE pipeline, determining exoplanet spectral lines (and elsewhere). Work from this collaboration was reported at Supercomputing 2013, held in Denver, November 2013. |
Start Year | 2013 |
Description | Intel/SGI/COSMOS Collaboration |
Organisation | Silicon Graphics Inc |
Country | United States |
Sector | Private |
PI Contribution | The present relationship is a longstanding collaboration and partnership in high performance computing with SGI since 1997 and with Intel since 2003. On the basis of taking delivery of the largest deployment of Xeon Phi processors in a single-image shared memory system, we have been testing the architecture and system stack while developing codes to exploit this new many-integrated core (MIC) technology. We influenced design of the initial "MG" blade accommodating the Xeon Phi within the SGI UV2, and we have worked with Intel to improve their Xeon Phi offload implementation. |
Collaborator Contribution | Our partner SGI has contributed half the salary GPB£18000 so that COSMOS could employ a parallel programmer, James Briggs. Intel have contributed 50% of an Intel employee, John Pennycook, to assist with development work and to act as a direct link to Intel software engineering teams. SGI has also contributed substantial software consultancy support through Cheng Liao which has led to new scalable SMP applications. |
Impact | There have been many positive outcomes from this collaboration, notably the Xeon Phi acceleration of important COSMOS consortium codes, including Camb (main CMB cosmological parameter estimation code), WALLS (early universe field theory code), MODAL (non-Gaussianity code), etc. A major development has been the implementation of a scalable shared-memory eigensolver capable of decomposing dense matrices up to 1M x 1M; this has important applications for the TROVE pipeline, determining exoplanet spectral lines (and elsewhere). Work from this collaboration was reported at Supercomputing 2013, held in Denver, November 2013. |
Start Year | 2013 |
Description | 10th Rencontres du Vietnam |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | 10th Rencontres du Vietnam Very High Energy Phenomena in the Universe Quy Nhon, Vietnam, 8 August 2014 Talk title: "Numerical simulations of coalescing binaries" |
Year(s) Of Engagement Activity | 2014 |
Description | 3rd Sant Cugat Forum on Astrophysics |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | 3rd Sant Cugat Forum on Astrophysics Sant Cugat, Spain, 25 April 2014 Talk title: "Gravitational Recoil and Astrophysical impact" |
Year(s) Of Engagement Activity | 2014 |
Description | 6th form talk (Chelmsford) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Schools |
Results and Impact | October 2017: Ulrich Sperhake gave a talk to a 6th form college in Chelmsford. Subject: The direct detection of gravitational waves by LIGO and the insights already obtained from this new window on the Universe as well as the enormous potential for our search for answers to the most profound questions in astrophysics and fundamental physics. |
Year(s) Of Engagement Activity | 2017 |
Description | Comenius |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | John D. Barrow gave four talks on cosmology to Comenius (Netherlands) University group, March and September 2014, with two more to come in November 2014 See above |
Year(s) Of Engagement Activity | 2014 |
Description | Foligno |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | John D. Barrow gave two talks at Foligno Science Festival, Italy, April 2014 See above |
Year(s) Of Engagement Activity | 2014 |
Description | IMO (Cape Town) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | IMO Celebrity Lecture, The Maths of Entire Universes, International Mathematics Olympiad, U Cape Town, 2014 The International Mathematical Olympiad (IMO) is the World Championship Mathematics Competition for High School students and is held annually in a different country. |
Year(s) Of Engagement Activity | 2014 |
Description | IOP Engineering Society (Chelmsford) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Professional Practitioners |
Results and Impact | October 2017: Ulrich Sperhake gave a talk to the IOP Engineering Society in Chelmsford. Subject: The direct detection of gravitational waves by LIGO and the insights already obtained from this new window on the Universe as well as the enormous potential for our search for answers to the most profound questions in astrophysics and fundamental physics. |
Year(s) Of Engagement Activity | 2017 |
Description | Lecture (Isaac Newton Institute) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Annual Correspondents Day Lecture, Isaac Newton Institute, Cambridge, 2014 The lecture was attended by representatives from EPSRC, BBSRC, ESRC, NERC and STFC. The lecture is important in maintaining and increasing the value of the Institute to the entire UK mathematical community |
Year(s) Of Engagement Activity | 2014 |
Description | New Perspectives on Cosmology |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Yi Wang gave a talk entitled "CMB Anomalies in the Light of BICEP2" at the IAS Workshop, "New Perspectives on Cosmology," 23 May 2014 at the Hong Kong University of Science and Technology During last few years we have witnessed rapid developments in cosmology, both in theory and experiments. Especially, with the recent PLANCK data our understanding on the universe is facing even more refinements than ever. Moreover, with the planned number of future experiments on the cosmic microwave background, large scale structure, gravitational waves and so on, our view on cosmology and gravity will be further developed continuously. The aim of the workshop was to promote stimulating informal and intensive discussions among the participants and to initiate possible collaborations. |
Year(s) Of Engagement Activity | 2014 |
URL | http://ias.ust.hk/workshop/cosmology/2014/index.html |
Description | Pordenone |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | John D. Barrow gave a talk and a formal interview at Pordenone Literary Festival, Italy, September 2014 See above |
Year(s) Of Engagement Activity | 2014 |
URL | http://www.oppureblog.com/john-barrow-ci-racconta-cose-linfinito/ |
Description | Rollins |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | John D. Barrow gave a talk at Rollins College Florida, "Origin and Evolution of the Universe," October 2014 See above |
Year(s) Of Engagement Activity | 2014 |
Description | Science Festival 2013 |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Public/other audiences |
Results and Impact | Substantial effort was made during the Cambridge Science Festival to publicise the role of the COSMOS supercomputer at the Centre for Mathematical Sciences, where hundreds of members of the general public visited on Saturday 23 March 2013. Group tours were offered every half hour which proved a particular highlight for the Open Day. The role of COSMOS for key science exploitation of the Planck satellite (announced on 21st March with considerable media coverage) was also emphasised. Considerable interest shown by members of the public. |
Year(s) Of Engagement Activity | 2013 |
Description | Vice Chancellor, Cape Town |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | John D. Barrow gave a talk in Vice Chancellor's series, University of Cape Town, "Origin and Evolution of the Universe," July 2014 See above |
Year(s) Of Engagement Activity | 2014 |